Universal Webpack or universal Babel

[sedubois]

As mentioned in #867 (comment) and #627 (comment), it sounds possible to use Webpack universally. There's e.g the universal-webpack package which might help, and this article gives some tips and an example.

If it can be done efficiently, having a universal Webpack sounds like a great way to reduce the complexity of the Next.js codebase and would allow users to load any Webpack plugin they need?

What's Next.js' team's opinion on this, does it make sense and should it be investigated?

[arunoda]

Having a such a system is nice. That's what we wanted ultimately.
But that solution seems like a bit of an experiment.

What we have today is a subset of Webpack but it's production ready.
And we've total control of how things works.

So, in the short-term we won't be able to hook into such system.

[sedubois]

Why would universal Webpack be more of an experiment than the Webpack + Babel combo?

Why couldn't we have total control of how things work, with a universal Webpack setup?

[davidrossjones]

Completely agree with @sedubois and @ccorcos (via #867).

I previously alluded to this in #658 and your linked comment above that mentions universal-webpack.

I too would love to see Next.js work with a full Webpack setup using target: node and pre-compiling on the server side. Much like is achieved via nuxt.js (a project inspired by Next.js) here: https://github.com/nuxt/nuxt.js/blob/master/lib/webpack/server.config.js

Babel plugins don't offer the same functionality as Webpack and I ❤️ SSR with Next.js so for me (and much of the community) this is a must have feature!

[sakulstra]

👍 then we could finally use libraries like which require us to import the css/scss manually without a hassle (e.g. https://github.com/Hardtack/react-material-components-web)

[ericclemmons]

I've been doing universal webpack apps successfully for well over a year now (I have a terrible sense of time, but it's been in production for over that long!), and here's a quick way of doing it with next:

// package.json scripts
"dev": "./node_modules/next/node_modules/.bin/webpack --config webpack.config.server.js --hide-modules --watch",

// webpack.config.server.js
require("source-map-support/register");

const CaseSensitivePathPlugin = require("case-sensitive-paths-webpack-plugin");
const FriendlyErrorsWebpackPlugin = require("friendly-errors-webpack-plugin");
const StartServerPlugin = require("start-server-webpack-plugin");
const webpack = require("webpack");
const nodeExternals = require("webpack-node-externals");

process.env.NODE_ENV = process.env.NODE_ENV || 'development'

const dev = process.env.NODE_ENV === "development"
const createCompiler = require("next/dist/server/build/webpack").default;

module.exports = createCompiler(".", { dev }).then((compiler) => {
  const server = Object.assign({}, compiler.options, {
    devtool: "inline-source-map",
    entry: {
      "server.js": [
        "webpack/hot/poll?1000",
        "./server.js",
      ],
    },
    externals: [
      /^@?\w[\w\-0-9\./]+$/,
    ],
    module: {
      rules: [
        {
          exclude: /node_modules/,
          loader: "babel-loader",
          options: {
            cacheDirectory: true,
            sourceMaps: dev ? 'both' : false,
          },
          test: /\.js(\?[^?]*)?$/,
        },
      ],
    },
    node: { __dirname: true, __filename: true },
    plugins: [
      new CaseSensitivePathPlugin(),
      new FriendlyErrorsWebpackPlugin(),
      new StartServerPlugin(),
      new webpack.HotModuleReplacementPlugin(),
      new webpack.NamedModulesPlugin(),
    ],
    target: "node",
  });

  return server;
});

From there I borrowed server.js from https://github.com/ericclemmons/webpack-hot-server-example/blob/master/src/server.js and used the custom-server-express example (minus the .listen call) and was off to the races!

[migueloller]

Just wanted to chime in and mention that if universal Webpack gets supported, it will fix issues such as #364 as well as reducing a lot of confusion around different behavior between client and server.

[kjs3]

Using webpack server-side would be especially nice for asset fingerprinting. This BUILD_ID PR not only lets you cache-bust assets, it forces cache-busting which is pretty heavy-handed. Typically, none of my assets change between deploys and when they do it's usually just a few files. Hashing the file keeps the fingerprint the same until bits change.

[rauchg]

There are a few issues with that @kjs3. If you bump the version of next let's say, the content hash of a page might remain the same, but the behavior would not. Therefore it needs to get invalidated.

[kjs3]

@rauchg, that's a good point as far as the js bundle is concerned.
In the webpack config you could use BUILD_ID for js/css file names but use [hash] for images.

[kjs3]

Actually, I think you'd use [hash] for css too.

[arunoda]

@kjs3 it's pretty nice idea to use webpack in the server as well. Then it gives us per assets hashing for FREE.

But the problem is how it works with complex apps. We wanted to make sure it won't use a lot of CPU cycles and so on. And it needs to work with our automatic code splitting feature as well.
And we need to support all the features including custom server and etc.

I think that's doable. But it require significant effort. (Specially testing)
So, we can't do it for 2.0, but we can have a look at after that and may the 3.0 be the all webpack version.

[arunoda]

@ide I hope you got some ideas on this too.

[ide]

I'd recommend looking into using Babel more than Webpack for SSR since Babel decouples preprocessing from bundling, in contrast with Webpack. Next's emit-file-loader is an effort to use Webpack without the "pack".

For sites with thousands of components I find Babel very appealing compared to Webpack, especially thinking about build times. Webpack can quickly get to 20 second build times and keeps growing to even a minute with sites much smaller than that size, while after the first build Babel can do much less work and run in under a second. (The site I'm working on has server-only code separate from client code so the server-only code never goes through Webpack, which works really well. Still, the client builds take a lot of time.)

We want to have the same plugins on the server and client, and Webpack is a means to that end, but from first principles I believe Babel plugins might be a better means to that end. This would be a new experimental direction: we could replace Webpack loaders and plugins with Babel plugins. These Babel plugins could generate & consume static resource metadata (similar to Webpack's build stats). They would work with a pure-Babel build for the server, and could also be invoked from babel-loader for the client-side Webpack build.

[sedubois]

@ide are there any production examples out there of "universal babel"? What is Babel's official position on the subject? Any documentation?

[migueloller]

@sedubois,

Babel is already used "universally" by Next.js. I think what @ide is suggesting is that there should be a shift in plugin reliance from Webpack to Babel to increase developer productivity. Because Webpack is designed to "pack" it comes with a lot of overhead that results in long build times.

Webpack is still needed to be able to take advantage of tree shaking and separate entry points but @ide makes a great point in that Webpack could be used just for that and then Babel would take care of the rest. In the end, client code would be Webpack'd (using babel-loader to use the new plugins) and server code would simply be transpiled with Babel.

I like this proposal!

[kjs3]

@ide, I'd love to see an example of asset imports being replaced with URLs the way url/file-loader works but with Babel alone. Because you're right, we don't really want the pack part of Webpack.

[ide]

@kjs3 Sure! This is a rough idea, a function that first processes all of the assets (ex: optimizes them, resizes them for different screen resolutions, renames them to be a hash of their contents) and produces a mapping from original asset name to renamed asset name (ex: "/static/icon.png" -> "/.next/static/914a48855b4.png"). Then this mapping is passed to a Babel plugin and Babel is invoked programmatically, this is the pseudocode:

async function buildAsync() {
  // Phase 1: process static resources
  let staticResourceMap = await preprocessStaticResourcesAsync();

  // Phase 2: compile JS for server
  for (let file in allSourceFiles) {
    babelCore.transform(file, {
      // bunch of .babelrc stuff here....
      plugins: [
        [require.resolve('./babel-plugin-static-resource-loader'), {
          // this is where webpack-like loader config ends up. probably want to offer some
          // defaults and also make it configurable
          {
             filenameTest: /\.txt$/,
             // For txt files we'll just naively export the text content
             async transformAsync(filename) {
               let contents = await fs.readFile(filename, 'utf8');
               return `module.exports = ${JSON.stringify(contents)};`;
             },
          },
          {
             filenameTest: /\.png$/,
             // For image files we'll export a link instead
             async transformAsync(filename) {
               // This is where we use the SR map!
               return `module.exports = ${staticResourceMap[filename]};`;
             },
          }
        }],
      ],
    });
  }
}

Taking this idea further, preprocessStaticResourcesAsync() could be based on Webpack (or something else, I worked on a static resource system for a complex website for a couple of years and believe there's so much more that can be done in open-source static resource bundling). After all we still need to create the client-side bundle with Webpack, and staticResourceMap can be based on the Webpack build stats.

In fact, this is structurally more robust than server-side Webpack today! One of the assumptions server-side Webpack often makes is that static resources will get renamed the same in the server-side Webpack config as they do in the client-side one. Ex: we need to ensure that both Webpack configs rename ./static/icon.png to /.next/static/914a48855b4.png or else the server-rendered markup won't match the client-rendered markup.

It would be nice if we could nix this content coupling, which is what the pseudocode does by taking the build stats (staticResourceMap) from the client-side Webpack build and uses it in the server-side build.

Configuration from application code

Furthermore, I anticipate wanting to have control over how assets are processed both in the build process and in the application code. So maybe implementing the AMD loader idea (which Webpack also supports) in these Babel plugins could be really good:

// No loaders prescribed by the application code; the build process decides how png files are handled
<img src={require('./static/icon.png')} />

// We want a custom loader!
// Gives you back some object like { width: 25, height: 25, colorProfile: 'srgb', ...etc }
let imageMetadata = require('image-metadata!./static/icon.png');

[migueloller]

@ide, this sounds like an interest project. Could even draw some inspiration from the recently released https://github.com/callstack-io/haul.

Also, https://github.com/istarkov/babel-plugin-webpack-loaders seems to want to do something like this. It has the added benefit that it aims to work with existing Webpack loaders.

The repo doesn't look very active but perhaps it could be contributed to or forked? It seems to be going in the right direction.

[ethanresnick]

Reading this thread, am I right to conclude that there's no (off-the-shelf) way to fingerprint assets in Next at the moment? If so, is there maybe an interim solution while the babel approach is being developed?

(More generally, I completely agree that webpack is sloooow, but replacing it with Babel seems like it will involve recreating a lot of stuff that already exists in the webpack ecosystem. Unless the Babel solution first runs static files through Webpack, as I now realize @ide's post suggests... that seems like a good idea.)

[sergiodxa]

@ethanresnick Next.js allow you to create a /static folder and place your assets there, then you can just use them as /static/your-image.png. For production usage it's better if you serve them from a CDN because Next.js set cache to 0.

[ethanresnick]

@sergiodxa Right. My question though is about how to put a fingerprint (hash of the files contents) in the file's name, and use that fingerprint in your server/client-rendered html. This fingerprint is useful for cache busting, regardless of where you serve your files from, so it's a separate concern.

[sergiodxa]

That's not possible right now, maybe you can add the hash to your files but you will need to change every usage of that file in your components.

Thinking about it you can dynamically generate a map of your assets paths (with hashes) and then import it in your Next.js code and use them.

[denisw]

Not being able to just add Webpack loaders and have them work on both the client and server is really painful.

We were evaluating a migration to Next.js for a project because it is a great fit on so many levels (effortless server-side rendering, the option to support static pages, etc.), but we simply cannot afford the multi-week effort of porting all of the external SCSS we have to styled-jsx or another CSS-in-JS solution. It would be so awesome if we could just add the Webpack loader we are currently using to a Next.js project and it would work on both the server and client, with hot reloading, imports and everything.

If universal Webpack would provide us with something like that, I'm all for it. The idea of making use of Babel for this is compelling, but then we are missing out on the mature Webpack loader ecosystem that already exists.

[stevefan1999-personal]

I give my support on making next.js language independent, the recently-released CoffeeScript 2.0 makes a powerful combination with next.js and React, but thanks to the hardcoded dependency with Babel on next.js, I cannot integrate CoffeeScript without an external transpiler process which is inconsistent and unpredictable (based on filesystem sync). This is also demonstrated in the example packages.

I'm also forseeing the possibility of integrating TypeScript with babel-preset-typescript but it has since still been alpha (for a whole year), and it makes it unsafe in production use.

Sorry if my wordings are kind of weird. I'm not a native user at all.

[jjant]

Has there been any progress with this? It's a pretty crucial feature to have in some cases.

I would love to help and make this possible, but I would need a few pointers on what the real issues are to just run webpack also for server code.

[belohlavek]

How can we help make this happen?