JS modules are loaded twice

[Reggino]

Bug report

Describe the bug

JS modules seem to be loaded twice

To Reproduce

Steps to reproduce the behavior:

1. Go to 'https://www.nextjs.org' (or your own built site)
2. Click on 'developer console' in e.g. Chrome
3. See Network tab
4. Do a hard refresh (E.g. CTRL + SHIFT + R)
5. Sort resources by name
6. Look at loaded JS modules. They appear twice in the resource list, but are not retrieved from cache

Expected behavior

All required resources to be loaded just once.

Screenshots

System information

• OS: Ubuntu Linux
• Browser Chrome
• Version of Next.js: 9.1.1 - 9.3.0

Additional context

When disabling granular loading in next.js config, the issue seems resolved.

    experimental: {
      granularChunks: false,
      css: false
    }

[Timer]

Can you turn off your experimental usage of the modern behavior and see if this persists?

[Reggino]

Can you turn off your experimental usage of the modern behavior and see if this persists?

The issue seems resolved using the configuration:

    experimental: {
      modern: false
    }

[Reggino]

(I've updated the title, since my assumption of the root cause may be incorrect ;-) )

[cansin]

I am having the exact same problem and do not have my experimental config turned on. According to the sourcemap, there are 2 places loading the JS files:

// at page-loader.js
.
.
.
function appendLink(href, rel, as) {
    return new Promise((res,rej,link)=>{
        link = document.createElement('link');
        link.crossOrigin = process.crossOrigin;
        link.href = href;
        link.rel = rel;
        if (as)
            link.as = as;
        link.onload = res;
        link.onerror = rej;
        document.head.appendChild(link); // <--- Here
    }
    );
}
.
.
.

and at document root (i.e. the HTML served) the actual <link rel="preload" ... /> tag. I am not sure when this started happening though. Ideas? I can try to create a small reprod repo if you want.

[cansin]

I am not sure if this is related but the reported First Load JS sizes are also roughly twice the size of First Load JS shared by all (which looks suspicious/weird):

Page                                                           Size     First Load JS
┌ λ /                                                          1.73 kB         227 kB
├   /_app                                                      5.92 kB         105 kB
├ ○ /404                                                       769 B           210 kB
├ λ /forgot-password                                           1.32 kB         227 kB
├ λ /login                                                     1.33 kB         227 kB
├ ○ /offline                                                   1 kB            211 kB
├ λ /reset-password                                            2.37 kB         228 kB
├ λ /signup                                                    1.31 kB         227 kB
└ λ /todos                                                     3.11 kB         228 kB
+ First Load JS shared by all                                  105 kB
  ├ static/pages/_app.js                                       5.92 kB
  ├ chunks/3704887d5176932cc40c267af232f229c1b773b4.e8a25d.js  41.7 kB
  ├ chunks/commons.f7f234.js                                   10.5 kB
  ├ chunks/framework.126679.js                                 40.4 kB
  ├ runtime/main.89a34b.js                                     6.02 kB
  └ runtime/webpack.c21266.js                                  794 B

[cansin]

And here is my next.config.js in case it helps:

const dotenv = require("dotenv");
const withSourceMaps = require("@zeit/next-source-maps");
const withSitemap = require("next-with-sitemap");

dotenv.config();

module.exports = withSourceMaps()(
  withSitemap({
    compress: true,
    env: {
      GQL_URL: process.env.GQL_URL,
      GQL_AUTHORIZATION: Buffer.from(
        `${process.env.GQL_CLIENT_ID}:${process.env.GQL_CLIENT_SECRET}`
      ).toString("base64"),
      GTM_ID: process.env.GTM_ID,
      GTM_AUTH: process.env.GTM_AUTH,
      GTM_PREVIEW: process.env.GTM_PREVIEW,
    },
    pageExtensions: ["page.js"],
    sitemap: {
      baseUrl: `${
        process.env.SECURE_SSL_REDIRECT !== "false" ? "https" : "http"
      }://${process.env.WEB_DOMAIN}`,
    },
  })
);

and some of my library versions:

    "@apollo/react-hooks": "^3.1.5",
    "@zeit/next-source-maps": "^0.0.4-canary.1",
    "next": "^9.3.6",
    "next-with-apollo": "^5.0.1",
    "react": "^16.13.1",
    "react-dom": "^16.13.1",

[cansin]

I want to speculate this issue might be related with #9070

[cansin]

I have also realized the issue does not happen for (Static) pages:

Which might further signal a hydration problem with SSR.

[cansin]

Huh, this is not happening on actual production, only happens when I run:

cansin@localhost web % export NODE_ENV=production
cansin@localhost web % yarn next build
cansin@localhost web % yarn next start

locally. So perhaps it is a false alarm? If so, I wonder what is the right way to simulate production locally.

[cansin]

Ok, I changed ports and back and this problem disappeared. Perhaps some kind of stale module hot-reload shenanigan was happening, despite trying to run a prod build. Sorry for the misdirection.

[Timer]

Closing this issue as the issue no longer reproduces on nextjs.org and these experimental features have been enabled by default for a while (minus modern), with no further reports.

[andrei-svistunou]

Hi @Timer ,

I still can reproduce this issue on a lot of sites, including nextjs.org

Could you, please, clarify this behaviour? Is it expected?

[andrei-svistunou]

Hi @Timer
Could you please take a look at my comment above? May we reopen the ticket or better to open a new one?

[helfi92]

We're seeing the same behavior as well on the latest version of Next.js 10.0.6

[SidOfc]

This seems to be happening in Chrome, or perhaps Chromium based browsers in general (didn't test that) in newer versions of next.js. In Firefox it seems like no request is duplicated. However, older versions of next.js don't seem to suffer from this issue even in Chrome.

For reference I'm testing this on Ubuntu 20.04 using Chrome 88.0.4324.182 and Firefox 86.0. Just specifying this in case it isn't reproducible on macOS / Windows, in which case it might be an issue with Chrome on Ubuntu.

Below some information with screenshots taken in Chrome.

I'm currently running next@9.5.5 to prevent this issue (see edit at the bottom). The repository for my website is public in case anyone is interested in running it locally. To reproduce the issue, simply change the version of "next" in package.json to "^10.0.7". Both requests shown here seem to be fired through the commons chunk in next.js 10+ if I am to believe the network inspector:

Additionally I did the same thing @andrei-svistunou did and can confirm that nextjs.org has this issue:

I've also tried setting the experimental options mentioned at the start of this issue to see if that would resolve it, I tried:

{
   experimental: {
        granularChunks: false,
        css: false,
        modern: false,
    }
}

but alas, no luck here either.

A friend and colleague of mine also has a next.js website using next@9.3.4, this does not show any duplicated requests. This also has a public repo for those interested.

Let me know if you want me to try anything to see if it resolves the issue and last but certainly not least, thanks for building such an awesome framework, I'm loving every bit of it!

EDIT

I've downgraded my own website to next@9.5.5 locally and the issue seems to have disappeared. My website went from loading ~500kb worth of resources to ~170kb on initial page load.

[Sheraff]

I was looking for the common denominator of all "duplicate" network calls that actually got through (weren't caught by the cache) and it seems that i found it: the first request is always coming from (index)

This means that the behavior we're seeing comes from the <link rel="preload" that are initially found in the HTML and not the <link rel="prefetch" added by the script afterwards.

The second request (the duplicate one) always comes from the same piece of code. It's the last line in the following function, packages/next/client/route-loader.ts, lines 91 - 115 (in the /master branch):

function prefetchViaDom(
  href: string,
  as: string,
  link?: HTMLLinkElement
): Promise<any> {
  return new Promise((res, rej) => {
    if (document.querySelector(`link[rel="prefetch"][href^="${href}"]`)) {
      return res()
    }

    link = document.createElement('link')

    // The order of property assignment here is intentional:
    if (as) link!.as = as
    link!.rel = `prefetch`
    link!.crossOrigin = process.env.__NEXT_CROSS_ORIGIN!
    link!.onload = res
    link!.onerror = rej

    // `href` should always be last:
    link!.href = href

    document.head.appendChild(link)
  })
}

My guess is that this function might need to extend its DOM selection to also look for link[rel="preload"] and not just prefetch. But to know for sure, I'd have to test a build with this change and I don't really have the time right now. Anyone who reproduces this issue on one of their current projects is willing to give it a go?

If so, just replace the selector from link[rel="prefetch"][href^="${href}"] to link[rel="prefetch"][href^="${href}"],link[rel="preload"][href^="${href}"].

---

To clarify, for all "duplicate" script requests that do get caught by the cache, they follow the normal principle of nextjs which is to be added to the DOM as <link rel="prefetch" (by the prefetchViaDom function pasted above) and then actually fetched by adding them to the DOM as <script src="...".

So this leaves me wondering why we don't see three copies of the same request in the case investigated above: 1 preload initially present, 1 prefetch added asap, and 1 regular script whenever needed. So maybe my grasp on the inner workings of nextjs is not very strong and you should take everything I said w/ a grain of salt!

[Sheraff]

@SidOfc If you want to test it out, the easiest way would be to search for link[rel="prefetch"][href^="${href}"] inside of your /node_modules/next folder and replace the string with link[rel="prefetch"][href^="${href}"],link[rel="preload"][href^="${href}"].

If you need a solution that lasts a little longer you could always point your package.json to my fork of nextjs instead of the official one, but I really don't recommend that long term (or even just in production for that matter) because it will be unmaintained.

[SidOfc]

@Sheraff alright, I did a search and found one occurrence in dist/client/route-loader.js which I replaced with the string you provided. After running a new build + export however, I'm still seeing duplicate requests:

I think the issue lies elsewhere, as I mentioned in my original comment (and can be seen in this screenshot) the requests seem to be coming from the commons chunk and neither of the types is index but rather script or javascript.

I hope this helps further narrowing it down a bit, if there's anything else you'd like me to try feel free to suggest, I think I can make any change necessary in the node_modules/next folder.

EDIT

Also noticing that css requests seem to be duplicated as well, the bottom two entries of my screenshot shows one request with type stylesheet and another with type fetch.

[Sheraff]

@SidOfc some duplication is normal and expected w/ nextjs. Some of it will appear from the preload or prefetch tags, and some of it will come from the actual requests that plan on using the resource. You should look at this with cache enabled and look at whether the resource is actually downloaded twice or just requested twice.

To give you an example, from vercel.com, this is what is normal:

And this is what we're trying to avoid:

In both cases, one of the requests comes from the prefetchViaDom function inside route-loader.js, that gets bundled into the commons chunk (this is where I asked you to replace the string). But this is just about prefetching the resource. Later on, a resource might be requested again but the request won't go through the network and be caught by the browser's cache which will answer with the prefetched resource (which is why you can't debug this w/ the "disable cache" devtools option toggled on).

[Sheraff]

And as for CSS, I had looked into it quickly myself, and couldn't find anywhere where it was an issue, since in all the cases of duplication I could find, the duplicate resource came from the cache. Here's an example of this from nextjs.org:

[SidOfc]

@Sheraff I see, indeed I did have cache disabled while testing this, thanks for pointing this out. After 1 reload I still noticed some requests were duplicated but after a second reload indeed all requests are correctly retrieved from prefetch/disk cache as seen in your screenshots.

The only thing I might add would be that this works flawlessly when using next start, however after running next export, if I use some other static file server such as serve and run serve out I do see duplicated CSS requests still. The reason for this seems to be that serve does not send appropiate caching headers:

serve

next start

I'm adding this just to further clarify why some resources might be loading twice in certain scenarios, Next.js has no control over how these static files get served so this is definitely not a Next.js issue.

Thank you for your time, effort, and explanation. It's much appreciated and clears up [my] confusion about how this works. I think I'll be switching back to next@10.0.7 now as that is the way forward :)

EDIT

P.S. is there any way to buy you a crate of beer or something?

[Sheraff]

@SidOfc Yeah I haven't tried setting cache headers w/ Next, my current project uses an express server (because it was started before next@10) and all cache headers are written from there.

I'm usually pretty happy w/ magic internet points, but there is always the buy me a coffee thing for special occasions.

[SidOfc]

More than happy to reward the kindness and constructiveness, enjoy 👍

[balazsorban44]

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.