Is it possible to use esbuild-plugin-pnp from the go build api?

[wegry]

I'm working on a Golang project that uses yarn 2's PnP for its front end. We're exploring a move to ESBuild.

ESBuild's plugin api is still experimental, but I'm at a loss for how to call esbuild-plugin-pnp from the Golang api.

https://esbuild.github.io/plugins/#using-plugins seems like you can't reference plugins directly.

[evanw]

It's deliberate that you can't call esbuild's JS plugins from Go and vice versa. I designed esbuild with the JS ecosystem in mind and it being written in Go was originally an internal implementation detail. I released the Go API because people requested it, but it's not way that I have designed JS ecosystem interop. IMO the best way to interop with the JS ecosystem is to use JS, otherwise JS libraries are extremely unergonomic to use.

However, it may still be possible to get this to work depending on how much effort you want to put in. Yarn has decided to make PnP a purely JS API so using it from Go is not easy. But it sounds like there is a limited way to call the Yarn 2 PnP API from other languages using stdin/stdout according to someone working on Yarn (#154 (comment)):

That being said, note that the .pnp.js file is also a valid standalone JS script ... if you run it without arguments, it'll spawn in daemon mode and you'll be able to interact with it through stdin/stdout there's a very basic protocol that allows calling the resolution and reading back the results that's how we implemented Flow PnP support, back in the day. Caveats: you still need to execute JS, in the end / resolution is a bit slower / you still need to add the zip layer support to the native tool (this is what was the final blocker with Flow).

So it's possible to port Yarn's plugin for esbuild to Go, but you'll have to invoke node as a subprocess to do it and you'll need to implement reading from a zip file yourself. You may also potentially need to know a fair bit about how Yarn 2 PnP works (I'm not sure since I've never used it myself).

[merceyz]

Yarn has decided to make PnP a purely JS API so using it from Go is not easy

It's possible to set pnpEnableInlining: false and Yarn will write the dependency info to a json file that you can parse, still need zip and virtual package support though.

[evanw]

Yarn will write the dependency info to a json file that you can parse

The generated file has this sentence at the top:

This file is automatically generated. Do not touch it, or risk your modifications being lost. We also recommend you not to read it either without using the @yarnpkg/pnp package, as the data layout is entirely unspecified and WILL change from a version to another.

So the data file is unfortunately unusable because the data format is volatile (i.e. cannot be relied upon) and cannot be read without running Yarn's JavaScript code. It would be a different situation if they decided on a data format and documented it but that's not what Yarn has done.

[rtsao]

PnP seems to a significant bottleneck when using esbuild at the moment.

At some point, I plan to implement PnP resolution as Go plugin. I would expect a substantial speedup from moving the zip reads into Go and caching pnpapi calls into JS. I think cache invalidation could be simple: just invalidate it anytime the .pnp.js file is modified. I think something along these lines would be a good stopgap until there is a more solid story around a language-agnostic PnP interface.

[evanw]

At some point, I plan to implement PnP resolution as Go plugin. I would expect a substantial speedup from moving the zip reads into Go and caching pnpapi calls into JS. I think cache invalidation could be simple: just invalidate it anytime the .pnp.js file is modified. I think something along these lines would be a good stopgap until there is a more solid story around a language-agnostic PnP interface.

This sounds like a solid approach to a Go plugin for Yarn 2 PnP.

I'm going to close this since the original question has been answered: no, it's not possible, but you can implement a Go plugin that does the same thing. See also #688 and #821.

[wegry]

Thanks for the quick response @evanw.

At least in our case, we ended up using nodeLinker: node-modules in our .yarnrc.yml to meet our "call ESBuild from Golang" needs. This project really is brilliant (and so fast!).

[evanw]

FYI: Native support for PnP has been added to esbuild: https://github.com/evanw/esbuild/releases/tag/v0.15.0 (it became possible to implement this now that the PnP data format has been documented: https://yarnpkg.com/advanced/pnp-spec/).