OOM when sourcemaps are enabled for data url loader

[stephen]

Hi,

I'm trying to get a relatively large (5k+ files) js/ts application previously using webpack to bundle and run with esbuild. I'm running the plugins branch at 139ae87

When I run esbuild without sourcemaps, things work great. The build takes ~2s and loads without problems. When I make the change to build sourcemaps, I see some non-deterministic behavior. One of two things happens:

1. Most of the time, esbuild will consume more and more memory until it gets OOM killed.

My machine has 64GB of memory. For more context, the output bundle from esbuild (in the non-sourcemaps case) is about ~30MB.

I grabbed a heap pprof here in case it helps: https://gist.githubusercontent.com/stephen/b187eb463e75cc342b5da94da68c0797/raw/c871fccb9673f676eb6cd1ed8284ebd1c7a172c6/heap.svg

2. Occasionally, I see the bundler complete in 3-4s with a big list of errors: Update: this seems unrelated

...
node_modules/react-data-grid/node_modules/@material-ui/icons/esm/index.js:3512:47: error: Could not read from file: /<snip>/node_modules/react-data-grid/node_modules/@material-ui/icons/esm/PermPhoneMsgRounded.js
export { default as PermPhoneMsgRounded } from './PermPhoneMsgRounded';
                                               ~~~~~~~~~~~~~~~~~~~~~~~
node_modules/react-data-grid/node_modules/@material-ui/icons/esm/index.js:4000:43: error: Could not read from file: /<snip>/node_modules/react-data-grid/node_modules/@material-ui/icons/esm/ReplyAllRounded.js
export { default as ReplyAllRounded } from './ReplyAllRounded';
                                           ~~~~~~~~~~~~~~~~~~~
5 warnings and 1388 errors

In this case, there aren't any other errors other than these Could not read from file. It seems like all errors from from the node_modules/react-data-grid/node_modules/@material-ui tree. I can confirm that these do exist:

$ ls /<snip>/node_modules/react-data-grid/node_modules/@material-ui/icons/esm/PermPhoneMsgRounded.js
/<snip>/node_modules/react-data-grid/node_modules/@material-ui/icons/esm/PermPhoneMsgRounded.js

Unfortunately, I can't share the codebase to show a repro, but I've been reading the source a bit and I'm happy to poke around more. My esbuild options look like this:

result := api.Build(api.BuildOptions{
	EntryPoints: []string{"<snip>"},
	Bundle:      true,
	LogLevel:    api.LogLevelInfo,
	Sourcemap:   api.SourceMapInline,
	Format:      api.FormatESModule,
	Loaders: map[string]api.Loader{
		".js":  api.LoaderJSX,
		".svg": api.LoaderDataURL,
		".jpg": api.LoaderDataURL,
		".png": api.LoaderDataURL,
		".gif": api.LoaderDataURL,
	},
	Defines: map[string]string{
		// snip
	},
	Plugins: []func(api.Plugin){
		// Both of these just discard/ignore .css and .graphql files.
		plugins.CSS,
		plugins.GraphQL,
	},
})

Do you have other suggestions for where to look in the esbuild code for what might be going wrong?

[stephen]

I ended up looking at a goroutine trace and seeing that things were getting stuck at quotedSource. It seems like the printer was choking on producing a sourcemap for image files that I had bundled with the data url. The problem goes away for me with this diff (which i'm definitely not suggesting):

--- a/github.com/evanw/esbuild/internal/printer/printer.go
+++ b/github.com/evanw/esbuild/internal/printer/printer.go
@@ -2855,6 +2861,13 @@ func quotedSources(tree *ast.AST, options *PrintOptions) []QuotedSource {
     return results
   }

+  if strings.HasSuffix(options.SourceForSourceMap.PrettyPath, ".svg") ||
+    strings.HasSuffix(options.SourceForSourceMap.PrettyPath, ".jpg") ||
+    strings.HasSuffix(options.SourceForSourceMap.PrettyPath, ".png") ||
+    strings.HasSuffix(options.SourceForSourceMap.PrettyPath, ".gif") {
+    return nil
+  }

How should these sources get opted-out of the sourcemap?

Separately, I think my deterministic failure is unrelated to this directly because I sometimes see it even with this fix, and in that scenario it doesn't even seem to get past the scan phase and into compile.

[evanw]

Thanks for pointing out the issue with the dataurl loader and binary files. I just fixed a lot of source map stuff in version 0.6.13 including binary files ending up in source maps. You can read more about what I fixed in the release notes.

Any chance your issue is fixed in the latest release? If not, is there any change in behavior? Does it still get stuck in the scan phase?

Also, since you're running on the plugins branch, are you using any plugins? Does this problem still happen with the publicly-released version?

[stephen]

Thanks for taking a look! I applied just 1e33ddb and it looks like it fixes my problem.

I also tried to upgrade to 0.6.13, but I'm also running into the panic from #311.

Also, since you're running on the plugins branch, are you using any plugins? Does this problem still happen with the publicly-released version?

I'm using plugins but the problem persists when I make them do nothing (just returning export default {} to the loader and not running other logic). It's a little hard for me to test the codebase without the commit altogether, since I'd have to go rip out a lot of imports to unsupported filetypes in the web app.