develop: esm support

[ryanio]

This PR adds ES modules support to develop, making it the primary module resolution method by adding "type": "module" to the package.jsons.

There is still some final work needed to complete all the pieces. Currently local tests are passing but karma is having some trouble, more details about karma in the bottom of this post.

This PR:

• Makes dist a container folder with folders inside: esm, cjs, and types
• Removes dist.browser build
  • We used to export separate browser builds to support ES5, but now that we are using native BigInts the browser builds also output ES2020, so there is no benefit or differentiation for a separate browser build.
• Adds eslint rule node/file-extension-in-import to automatically add .js file extensions (really nice to have).
• A couple node_modules have hotfix workarounds on postinstall scripts. For the ones I haven't opened issues on the respective repositories, would be nice to eventually do so the hotfixes can be removed in the future.
• Sets esModuleInterop and allowSyntheticDefaultImports back to false (see here and Unnecessary enforcing esModuleInterop for users of the package  #1615)

package.json exports now looks like this:

  "main": "./dist/cjs/index.js",
  "types": "./dist/types/index.d.ts",
  "browser": "./dist/esm/index.js",
  "exports": {
    ".": {
      "import": "./dist/esm/index.js",
      "require": "./dist/cjs/index.js",
      "types": "./dist/types/index.d.ts",
      "browser": "./dist/esm/index.js"
    },
    "./*": {
      "import": "./dist/esm/*",
      "require": "./dist/cjs/*",
      "types": "./dist/types/*",
      "browser": "./dist/esm/*"
    }
  },

this allows for top-level exports, e.g.:

import VM from '@ethereumjs/vm
import Common, { Chain } from '@ethereumjs/common'

while also providing deep-access to files inside in case they are not exported at the top level, e.g.:

import Bloom from '@ethereumjs/vm/bloom/index.js'
import { GenesisState } from '@ethereumjs/blockchain/genesisStates/index.js'

---

I haven't been able to get the commonjs typescript build working yet since it complains about some of the esm changes, so one approach might be to instead build the commonjs version with webpack from the esm code. Additionally single file browser bundles could be added so others can easily use these packages from the browser:

  "exports": {
    ...
    "./esm-browser-bundle": "./dist/bundles/esm.js",
    "./iife-browser-bundle": "./dist/bundles/iife.js",
    "./umd-browser-bundle": "./dist/bundles/umd.js",
  }

@MicahZoltu shared his thoughts about bundling in discord:

I generally recommend against libraries doing bundling. Bundling should be done (IMO) at the application layer.
Under this paradigm, you would publish ESM and CJS builds of your library, and you wouldn't bundle in transitive dependencies.
The caveat here is that if you have any dependencies that don't provide ESM builds then things get complicated.
Ideally, you just don't have any non-ESM dependencies.
And if there is some dependency that you really need which doesn't have an ESM build you would submit a PR to add an ESM build and make the world a better place for everyone. 😀

I would agree here to keep our builds light and modern for the esm future without the burden of compiling the transitive dependencies. This would be aligned with our current strategy (on master branch) to let application layer do the bundling, since each set of needs could be different.

karma

I haven't been able to get karma fully working yet, getting this error on start: SyntaxError: import declarations may only appear at top level of a module. If we export browser bundles with e.g. webpack, then it might make sense to switch from karma-typescript to karma-webpack for better e2e testing.

(I only suggest webpack because we already use it in the client. If rollup is a better experience, then that can also be used and the client build can also be switched over to it to share the same deps.)

[ryanio]

Some more digging into karma, I thought these acorn options might help but no luck so far. Still getting Uncaught SyntaxError: Cannot use import statement outside a module. I suspect it's routing it through commonjs bundling instead of trying to use esm.

        acornOptions: {
          ecmaVersion: 11,
          sourceType: 'module',
          allowImportExportEverywhere: true,
        },

There's some lengthy information about the bundling here https://github.com/monounity/karma-typescript/tree/master/packages/karma-typescript#automatic-bundling and it says

However, this intentional behavior makes it possible to use karma-typescript for projects that use the Typescript module system, or have the compiler option module set to another value than commonjs, or simply put everything on the global scope.

So this should be supported.

[MicahZoltu]

Why are the types separated from the module output? One issue with this is that the type definitions for cjs may not be the same as the type definitions for esm, so they generally shouldn't share type definition files. Also, it is a defacto standard to include TS definitions next to the output files and source maps, and following that defacto standard is more likely to result in stuff "just working" without the user needing to do additional work.

[MicahZoltu]

Recommend declarationMap: true as well (https://www.typescriptlang.org/tsconfig#declarationMap).

[MicahZoltu]

Rather than doing this, you could use a compiler plugin to automatically do it. There is much disagreement on what is "correct" here, but I'm of the opinion that you should leave ./header and let the compiler add the extension. The reason for this is because you aren't actually importing ./header.js, you are importing ./header.ts and then after transpilation it should turn into ./header.js.

If you want to have the compiler automatically add the extension, rather than adding them manually, you can use the extension here:
https://github.com/Zoltu/typescript-transformer-append-js-extension/

[holgerd77]

Yes, I also have to say that I am still battling hard with accepting that this (the .js file extension) needs to be added (and I am not sure how this battle will play out 😋, since I am strongly feel this inconsistency Micah mentioned, so having these "wrong references" in our whole code base and TypeScript somewhat generously "looking over it" (does it?)).

So can we eventually give this extension a try?

[holgerd77]

Why are the types separated from the module output? One issue with this is that the type definitions for cjs may not be the same as the type definitions for esm, so they generally shouldn't share type definition files. Also, it is a defacto standard to include TS definitions next to the output files and source maps, and following that defacto standard is more likely to result in stuff "just working" without the user needing to do additional work.

@MicahZoltu Could you very concretely outline what this would specifically mean so that we make sure that we are on the same page? Not using the types location reference in package.json? 🤔

[holgerd77]

Some procedural note: this is still so highly invasive that I am not sure if we can merge this in early respectively "in time", so I would consider to not include this in the first round of beta releases for the breaking releases.

Since other work is waiting on a new develop branch basis I would still continue and replace develop with the rebased version once #1928 is ready, hope this is not causing too much hazzle regarding this PR.

[acolytec3]

I've started poking around in this PR and I'm seeing a lot of errors I don't really understand trying to build it so not sure if that's what @ryanio was referring to by the CJS stuff not working but I'm guessing it'll take me some time to wrap my arms around it.

[acolytec3]

I will also say I don't love the idea of adding a webpack build step to every one of our libraries. If the build step is half as slow as it is for our client build step at present, it will likely greatly increase the time it takes for an npm run clean && npm i refresh of the repo (which we have to do somewhat regularly at present).

[MicahZoltu]

@MicahZoltu Could you very concretely outline what this would specifically mean so that we make sure that we are on the same page? Not using the types location reference in package.json? 🤔

The fix I am proposing would just involve removing the types keys from package.json and removing declarationDir key from tsconfig.json.

[ryanio]

i've separated out tasks from this PR to #1946, will close this PR for now, @MicahZoltu pls take a look at the issue and feel free to suggest anything else in there.