RFC: automatically transpile files based on baseUrl configuration

[jeantil]

• Related PR

  • improves baseUrl resolution in typescript monorepos #13542 (merged - fully resolves tsconfig including extends and resolves ts path aliases relative to baseUrl automatically)
  • enables transpilation for packages below baseUrl #15569 (pending - automatically enables transpilation for modules which are resolved through path aliases/baseUrl)

• Related issues:

  • jsx file doesnt like jsx? #15251
  • Failure to resolve modules from yarn workspaces when module has rootDir/outDir #13197
  • tsconfig rewrite not respecting extends #13616
  • Cannot import Typescript files from outside of 'dir' directory #9474

• RFC

  • Initially PR improves baseUrl resolution in typescript monorepos #13542 was written to solve Failure to resolve modules from yarn workspaces when module has rootDir/outDir #13197 and make it easier to build next projects which are a module within a monorepo. There were two issues that needed to be fixed: properly resolving modules internal to the monorepo and transpiling the files in the internally resolved modules since such modules are built alongside the next app and are not necessarily destined to be published in a normal manner.

  • Solving the module resolution depended in resolving the full typescript config and using that to build the proper aliases and was fixed by improves baseUrl resolution in typescript monorepos #13542.

  • Transpiling internal modules requires changing the babel-loader configuration to include paths which are outside of the directory which depends on next.

  • Current situation
    Without improves baseUrl resolution in typescript monorepos #13542 files in monorepo modules a next project depends on, would not be resolved correctly, naively adding a nextjs app in a monorepo results in a cryptic resolution failure even though the documentation states that path aliases are supported and automatically derived from tsconfig

    With improves baseUrl resolution in typescript monorepos #13542 the modules are correctly resolved but they are not transpiled, so in a project with the following structure

     .
    ├── node_modules
    ├── packages
    │   ├── lib
    │   │   ├── a
    │   │   │   └── api.ts
    │   │   └── b
    │   │       ├── api.ts
    │   │       └── b-only.ts
    │   └── www
    │       ├── components
    │       │   ├── alias-to-d-ts.d.ts
    │       │   ├── alias-to-d-ts.tsx
    │       │   ├── hello.tsx
    │       │   └── world.tsx
    │       ├── next.config.js
    │       ├── next-env.d.ts
    │       ├── pages
    │       │   ├── alias-to-d-ts.tsx
    │       │   ├── basic-alias.tsx
    │       │   ├── resolve-fallback.tsx
    │       │   ├── resolve-order.tsx
    │       │   └── single-alias.tsx
    │       ├── test
    │       │   └── index.test.js
    │       └── tsconfig.json
    └── tsconfig.json
    

    Typescript or javascript files in packages/www will automatically be transpiled but files in packages/lib won't which I found surprising.

• Proposal
  Automatically including the configured baseUrl in the transpilation root of next-babel-loader would completely remove the need for a custom next.config.js and automatically transpile files resolved within the same monorepo through path alises.

[ghost]

Great work diving into this all! My project structure is a bit different so wanted to share. You've dug into this a ton more than I have so I don't know how helpful I can be aside from saying that I would love to see Next natively support this (without next-transpile-modules).

My project:

├── apps
│   ├── one
│   │   ├── next-env.d.ts
│   │   ├── next.config.js
│   │   ├── package.json
│   │   ├── src
│   │   │   ├── components
│   │   │   └── pages
│   │   │       └── index.tsx
│   │   └── tsconfig.json
│   └── two
│       ├── next-env.d.ts
│       ├── next.config.js
│       ├── package.json
│       ├── src
│       │   ├── components
│       │   └── pages
│       │       └── index.tsx
│       └── tsconfig.json
└── packages
    ├── a
    │   ├── index.tsx
    │   └── package.json
    └── b
        ├── index.tsx
        └── package.json

I deploy each Next app separately (will be over a dozen for now), packages are published to npm separately, and each Next tsconfig is extended from one of the packages (as a util).

I don't use baseUrl because my packages are published and imported via the package name without aliases. Because of how yarn workspaces resolves packages, I decided to not use /dist folders on the package builds and added a pre-publish script to update my package.json main. On dev, I can resolve, but not transpile so I use next-transpile-modules. Curious how your proposal would work for my scenario. I'm open to adjusting my project structure if there's a better way, but as far as I remember baseUrl didn't work for me.

[jeantil]

I deploy each Next app separately (will be over a dozen for now), packages are published to npm separately, and each Next tsconfig is extended from one of the packages (as a util).

Since you don't use baseUrl the change I propose here wouldn't affect you one bit. The change only affects people who use baseUrl along with path aliases.

On dev, I can resolve, but not transpile so I use next-transpile-modules

Since we use baseUrl with path aliases resolving requires a slightly more involved setup which is solved without having to add external tooling in #13542.
Having resolution work out of the box with path aliases is nice but then we get into the transpilation issue same as you. My initial PR in #13542 allowed to automatically transpile files from any package resolved through baseUrl. Nextjs team felt that this was too much of a change and requested this RFC before moving forward with it.
In your case since you don't use next-transpile-modules, it wouldn't change anything at all. But we share packages between nextjs apps and pure node apps which have different packaging requirements (es target and module types) so we do use a dist directory for the output of our compiled packages, but we use path aliases for ts-node development and for nextjs it can be made to work but could be automatically supported by nextjs.

Also next-transpile-module is a great tool but the author only follows stable next versions and we have been stuck before waiting on a fix that was only available in a canary version.

I'm open to adjusting my project structure if there's a better way,

you wouldn't have to change a thing to your project structure, and unless you wanted to use baseUrl and path aliases, the change would be totally transparent for you. if you use baseUrl the change would allow you to simplify your tooling setup.

[ghost]

Everything is OK. Dev is ideal right now because yarn workspaces symlinks to the monorepo packages. Next dev is pulling in source files to the apps so the entire monorepo is hot. I did this so I can do testing with all package components and tweak packages without running a separate build/watch task. Publishing is done via Actions and orchestrated before deploys. My only concern right now is my use of next-transpile-modules in all of my next apps. Do you have any links to projects that use a baseUrl structure similar to what you propose? I checked many of the links above but there wasn’t anything very complex like what I have.

[jeantil]

Do you have any links to projects that use a baseUrl structure similar to what you propose?

No I don't, my primary project which is much more involved than the reproduction samples I provided is unfortunately not open source.

[jeantil]

My only concern right now is my use of next-transpile-modules in all of my next apps.

@jasonbarone you should be able to replace next-transpile-modules with next's own babel transpiler, look at https://github.com/vercel/next.js/blob/canary/test/integration/typescript-workspaces-paths/packages/www/next.config.js for an example

[ghost]

Ah ok no worries, thanks. I will give your PR a try this week and do some testing with my configuration.

[jeantil]

I'm not sure what you mean with vercel's new monorepo support, I was on vacation and have yet to find out all that they implemented :)
I was able to deploy my sample app https://github.com/jeantil/next-9-ts-aliases-workspaces/ at https://vercel.com/jeantil/next-9-ts-aliases-workspaces/3yd8k62ao/source without any obvious issues

here are my settings in vercel

yarn build delegates to building the web-app

    "build": "yarn --cwd packages/web-app build",

does that help ?

[ghost]

Ah I see, in your example you're deploying the entire project repo and using commands from the root. The "monorepo support" currently only support the Github integration and the Root Directory setting. If you connect your repo to the Vercel Github integration and specify packages/web-app in the Root Directory above, Vercel will only deploy the web-app subdirectory, which would cause some of the issues I mentioned.

[jeantil]

well I would have to investigate more but that's a vercel specific issue, not really related to next :) do you have a public deployment I could look at ?

[ghost]

I don't have a deployment, but here's a repo with the same setup: https://github.com/jasonbarone/next-typescript-monorepo-test. To clarify this example: this is pretty much one of the best setups I've found to meet my requirements below. It does require next-transpile-modules or else the local packages will fail in the Next.js apps.

1. All Next apps can resolve/transpile local packages without any additional compile/transpile processes. In other words, I need to be able to spin up any single Next app and I can code anything in the monorepo without running anything other than next dev in the app I'm working on.
2. All packages should be published to Github/NPM in order to allow the Next apps to be deployed via their respective project folders via Vercel.

Notes:

• tsconfig baseUrl cannot be used because the Next apps must be deployed via their project folders.
• tsconfig extends from the monorepo root cannot be used for same reason.

I realize that my repo setup is not really related to your baseUrl PR and RFC, but just wanted to provide it as input regardless.

[belgattitude]

@jasonbarone

Notes:
tsconfig baseUrl cannot be used because the Next apps must be deployed via their project folders.
tsconfig extends from the monorepo root cannot be used for same reason.

This has been fixed, the the last bug I've runned into is with api-routes... they are working on it: #16667 (comment).

[Dashue]

What's the status of this?
I'm currently looking to add an expo app and want to re-use some of my nextjs functionality and it seems that the approach enabled in this RFC would save me a lot of reading up and hacking around?

[belgattitude]

@jeantil I'm experimenting with the baseUrl approach and produced a working example repo.

https://github.com/belgattitude/next-transpile-ts-workspace

I would like to propose an official example for next

Do you mind to have a look ?

[ghost]

Nice example, it's worth calling out is you can test the Vercel monorepo support by toggling it on here: https://vercel.com/_flags. I'm using an example similar to yours and the only thing that stood out as a problem was the inability to deploy to Vercel because the tsconfig is extended from the root.

[belgattitude]

@jasonbarone @jeantil the monorepo is official now: https://vercel.com/blog. There's still an issue with api-routes...

I've set up another repo: https://github.com/belgattitude/vercel-monorepo-test with deployment examples and a mini readme.

The only drawback I have from next-transpile-module is the css/scss support. I'll try to setup an hybrid solution, but it's very encouraging.

[jeantil]

I'm on vacation at the moment, I'll have a look next week when I'm back. If you look at the referenced PRs there is an integration test which demonstrates a monorepo with both baseUrl ( first PR merged) and baseUrl + automatic transpilation (second PR not merged yet) of code below baseUrl Regards Le mar. 18 août 2020 à 08:55, Sébastien Vanvelthem <notifications@github.com> a écrit :

…

@jeantil <https://github.com/jeantil> I'm experimenting with the baseUrl approach and produced a working example repo. https://github.com/belgattitude/next-transpile-ts-workspace I would like to propose an official example for next Do you mind to have a look ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15327 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFTQZC36R3BFPWWJXW7DDSBIQWBANCNFSM4PCK3NPA> .

[jeantil]

however I'm seeing some issues when using with Vercel's new monorepo support: - Extending a tsconfig from the project root will fail on Vercel due to not being able to resolve the config in the cwd above the project folder. - Tsconfig paths set in the nested Next project's tsconfig do not work when deploying, I assume because of similar reason. @jeantil <https://github.com/jeantil>, did you run into this?

I'm not sure, for me it didn't work locally either, ence the 2nd PR. Without specific configuration, next transpiles typescript code using babel, but only if the code is in the next.config.js directory or in a subdirectory of that. When using typescript paths in a monorepo you will usually extend a tsconfig which is in a parent directory of the directory which contains next.config.js in which case baseUrl will most likely also resolve to a parent directory. When webpack resolves the files for your imports it will correctly resolve the path to the typescript files but the babel transpilation is restricted to paths that are below next.config.js and webpack fails to load the typescript files since they are not transpiler. If you merged the pr it will work locally but to get it to work on vercel you will have to publish your own version of next since the second part of the fix is not in official builds —

…

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15327 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFTQ4FGRVRCQADT7YG4BDSBKNDRANCNFSM4PCK3NPA> .

[Dashue]

Is the nextjs team aware of this thread?
They don't seem to be participating and I find them to be normally be very responsive, not having input for 30d seems odd?

[jeantil]

This discussion was opened as a followup to #13542 where @timneutkens asked to split automatic transpilation configuration from the path resolution. So I'm pretty sure that he is aware that the discussion exists. I assume that they have a lot of subjects to handle and mono-repo is not very high on the list :)

[belgattitude]

@jeantil I'm really interested in this RFC, haven't really participated yet cause I need to experiment first.

I researched a bit pros and cons:

Support matrix	tsconfig paths	next-transpile-module
Typescript	✅	✅
Javascript	✅	✅
NextJs Fast refresh	✅	✅
CSS	❌	✅
SCSS	❌	✅
CSS-in-JS	✅	✅
ts-jest	✅	?
Vercel monorepo	✅ api-routes coming !	✅
Yarn 2 PNP	✅	❌
Experimental webpack5	?	❌ (working on it)

I'll need to test it out more, but do you think it would be possible to include css/scss support as well ?

My test repo is here: https://github.com/belgattitude/vercel-monorepo-test

[jeantil]

We use styled-jsx and our styles are defined in the js/jsx/ts/tsx files of their components so I didn't dig into plugins we don't use ...

My understanding of all this remains fairly limited but from what I understand and in a very simplified view there are 2 steps that are needed :

1. resolving imports/requires to actual files <= that's where paths come into play
2. transpiling the content of the files <= that's where my PR comes in for test: /\.(tsx|ts|js|mjs|jsx)$/,

With regards to css/scss I have pretty much no idea how these plugins actually accomplish step 1, it may be that the current tsconfig is enough for them to find the .module.css or .module.scss files though this line is suspicious (dir is the toplevel directory of the next project which in a monorepo is below baseUrl in the directory tree).
Then you would need to make sure that the corresponding transpilation plugin is actually summoned for these files.

In my undestanding the reason why ts files in a monorepo were not being transpiled was that the path to the file was not part of the directories included in the babel transpiler configuration. All my change does is enlarge the subtree that is being considered by the babel transpilation plugin so that any file resolved under baseUrl is transpiled.

This can actually be done manually: you can add a second occurence of the babel plugin and configure it to process all files under baseUrl (see the integration test in my first PR for the details)
I guess you can look at how the css/scss plugins are configured in https://github.com/vercel/next.js/blob/canary/packages/next/build/webpack-config.ts
the first step would be to come up with a manual configuration which extends this configuration and see if it's enough for it to process the files, then it would probably be quite simple to extend the PR to include this configuration in the base config.

[matamatanot]

@timneutkens @jeantil
Is there any progress on this RFC?
I would love to have this feature to manage multiple Next.js projects.

[amlcodes]

Is there any progress with this? With Vercel acquiring turborepo, seems extremely necessary.

[belgattitude]

@amlcodes Is the new experimental flag in netxtjs 10.2+ does solve your issue ?

• Original P/R: Experimental feature for allowing importing Typescript files outside of the root directory #22867
• Support thread: Experimental TypeScript compiling outside of app directory feedback thread #26420

Just add in your next.config.js

const nextConfig = {
  experimental: {
    externalDir: true,
  },
};
export default nextConfig;

Working example here: https://github.com/belgattitude/nextjs-monorepo-example

[bobaaaaa]

@amlcodes did you try https://github.com/martpie/next-transpile-modules? This works "okay" for our setup.

[amlcodes]

Yes. I should have also said that including next-transpile-modules in our next.config.js, or not including it, along with or without externalDir: true - doesnt help

[belgattitude]

Can you post your tsconfig ?

It seems the alias should be @/components/*: ["../../packages/ui/components/*"] (../.. might depend on baseUrl)

Example here: https://github.com/belgattitude/nextjs-monorepo-example/blob/main/apps/web-app/tsconfig.json#L21

[bobaaaaa]

We also removed aliases for packages. So instead of @/components, we use the regular package name like @scope/components. With that, you only have to deal with the typescript path problem.

[belgattitude]

Is far better yes. I would recommend that too