New Assets Build Tool (Assets Manager)

[Skrypt]

Some extra work for me on Christmas holidays.🎅🏼

Assets Manager

Created by @jptissot

Based on Concurrently and Parcel but can also run Vite and Webpack too. This is a non-opiniated build tool which allows to be extended for any asset compiler/bundler that someone may require.

This build tool essentially uses Concurrently instead of Gulp.
Concurrently, is a concurrent shell runner which allows to trigger any possible shell command. It allows to trigger Parcel, Vite, Gulp or any other command that we need for building assets. Everything is written as ES6 modules (.mjs files).

Kind of needed for Vue 3 migration because it needs to use either Vite or Parcel to build as ES6 modules.

Old assets are not compiled as ES6 modules so they don't need these bundlers. For that matter I kept the old gulpfile.js which will be now triggered by Concurrently.

What needs to be done over time is to migrate these javascript files to ES6 modules (.mjs files or .ts files) so that we can compile them as modules. But that's a migration that will happen over time.

Features

• Build everything: yarn build
• Build module by name: yarn build -n module-name
• Watch module by name: yarn watch -n module-name.
• Build with gulp: yarn build -g
• Rebuild with gulp: yarn build -gr
• Host with dev server: yarn host -n module-name.
• Clean folders with yarn clean. Will also clean parcel-cache folder.
• Makes uses of latest yarn version 4.6.0. OC is currently on version 1.something
• Makes use of yarn workspaces which allows to import files from different locations in the app for sharing ES6 modules.
• VS Code launcher debug option added as "Asset Bundler Tool Debug"
• Build assets by tag: yarn build -t tagname
• Gulp pipeline still using GulpAssets.json file
• New Assets.json file definitions for building with new tool.
• Concurrently will retry building up to 3 times making CI build less prone to fail.
• Moving package.json files to Assets folder of modules and themes. This will remove these files from the compiled C# .dll files.
• Removes unnecessary package-lock.json files as we now rely on a single yarn.lock file.
• When using watch sass action it will recursively watch imports in files.

Update

Finally, I'm keeping the old gulp pipeline because there is nothing wrong with it. Also, for backward compatibility with older modules that requires it. I'm using GulpAssets.json for the Gulp build tool and using Assets.json for the new one. This way, no need to migrate to the new build tool. I'm simply triggering gulp rebuild with Concurrently in the new asset build tool. Because that's what it is essentially, a concurrent shell runner. This way, it is going to be a softer migration for those who already have modules that are built with the old gulp pipeline.

Of course, this PR needs to be accepted by community first.

TODO

• More documentation.

Modules to migrate:

• OrchardCore.Resources (Almost everything worked except (Monaco and Trumbowyg RTL)
• OrchardCore.Media (Not a great idea to waste time on migrating this as we are refactoring to Vue 3)
• OrchardCore.Liquid (Monaco Editor Intellisense)

@jptissot @deanmarcussen @aderderian

Fixes #14169. Related to #15740.

[Piedone]

You've been sneakily working on this since April ;).

Fixes #15740 #14169 #6762?

[hishamco]

@Skrypt is there a need to remove the old node_modules folder?

[Skrypt]

Yes, you do a gît clean -xdf to clean everything

[hishamco]

I expect that, thanks

[gvkries]

@Skrypt When I do corepack enable it gives me the following error:

Internal Error: EPERM: operation not permitted, open 'C:\Program Files\nodejs\pnpm'
Error: EPERM: operation not permitted, open 'C:\Program Files\nodejs\pnpm'

Is that expected, e.g. if corepack is enabled? yarn worked anyway on my machine.

[Skrypt]

I think the issue is that pnpm is installed globally on your machine.
You probably did npm install pnpm -g or something similar.

You will need to uninstall it before and let Corepack install it per "project folder".
npm uninstall pnpm -g

[gvkries]

On that machine I only installed Visual Studio and Node. As corepack seems to be available automatically, I was thinking corepack enable may not be required all the time. But doesn't matter much, thank you.

[Skrypt]

Yes, you need to execute corepack enable on the root folder of your OC repository. It is always required when the packageManager param is set on a package.json file. Else, it won't automatically install/use/update to the proper version of that package manager.

But the strange part is that it is failing on pnpm which is only used by OrchardCore.Commerce. We use Yarn with OC. Unless you are trying to make this asset manager work with OC.Commerce I don't see why it is failing on pnpm.

[Skrypt]

Also, you need to stop using NPM with OC. You need to start using Yarn equivalent commands else it will add unwanted package-lock.json files everywhere.

And from an interesting stackoverflow page they seems to suggest to do also:

npm cache clean --force

To fix the issue.

[gvkries]

I forgot that I've used npm install -g corepack.

[gvkries]

npm cache clean --force

That doesn't fix the error for me. Note that yarn build -gr doesn't give me any error.

[gvkries]

I don't really understand how all this Node stuff is working, but I can run corepack enable only in the actual install folder of npm (in %AppData%\npm on Windows). That has changed the batch files for yarn, pnpm and others.
Looks like it doesn't have to be run in the OC root?

[Skrypt]

Have you tried using a terminal with administrator privilege's?
It has to be run in the OC root to get the same fetched yarn dependencies.
If you don't enable Corepack then it will use yarn version 1.1.something which is what Node.js installs by default.

Corepack has been introduced to allow to manage package managers for those who uses multiple versions of them for different projects. It becomes pretty much unmanageable over time if you have projects that uses Yarn and pnpm and that these projects only compiles with a specific version of them.

Here, by not using Corepack and the proper version of Yarn it will install ; we can't predict if it will compile exactly the same as the CI does. Which will eventually leads into issues when submitting PR's. But not only that, but you want to have reproducible builds to avoid having issues from having a different compilation outcome.

[gvkries]

Thank you for your detailed help.

The file 'C:\Program Files\nodejs\pnpm' didn't exist on my machine, it's in '%appdata%\npm'. If I run the command with admin permissions, it will create the same batch files in the program files folder, but I don't see anything related to the working folder (the OC root folder in our case). Looks the same for me as when I was using corepack in the npm folder, as both directories are in the path environment variable.

Anyway, it's working as described. Maybe we should add a note about admin permissions on Windows.

[MikeAlhayek]

@Skrypt this PR introduced a problem. By default it is not generating the .min.js file. At least TheAdmin-main.min.js and TheAdmin.min.js are missing from the project and are causing production issues.

How to generate these file using the new setup?

[MikeAlhayek]

I created issue #17522 for tracking

[gvkries]

yarn build -gr is now suddenly not working anymore, without changing anything. There are invalid paths in some copy commands:

[theme-theblogtheme] Error copying (from, to) C:/Users/georg/source/repos/OrchardCMS/OrchardCore/src/OrchardCore.Themes/TheBlogTheme/Assets/TheBlogTheme/dist/assets/favicon.ico 
c:\Users\georg\source\repos\OrchardCMS\OrchardCore\src\OrchardCore.Themes\TheBlogTheme\wwwroot\assets\C:\Users\georg\source\repos\OrchardCMS\OrchardCore\src\OrchardCore.Themes\TheBlogTheme\Assets\TheBlogTheme\dist\assets\favicon.ico 
Error: Path contains invalid characters: 
c:\Users\georg\source\repos\OrchardCMS\OrchardCore\src\OrchardCore.Themes\TheBlogTheme\wwwroot\assets\C:\Users\georg\source\repos\OrchardCMS\OrchardCore\src\OrchardCore.Themes\TheBlogTheme\Assets\TheBlogTheme\dist\assets

[Skrypt]

I'm looking at your path there and it is:

c:\Users\georg\source\repos\OrchardCMS\OrchardCore\src\OrchardCore.Themes\TheBlogTheme\wwwroot\assets\C:\Users\georg\source\repos\OrchardCMS\OrchardCore\src\OrchardCore.Themes\TheBlogTheme\Assets\TheBlogTheme\dist\assets

Which is 2 distinct paths:

c:\Users\georg\source\repos\OrchardCMS\OrchardCore\src\OrchardCore.Themes\TheBlogTheme\wwwroot\assets\

C:\Users\georg\source\repos\OrchardCMS\OrchardCore\src\OrchardCore.Themes\TheBlogTheme\Assets\TheBlogTheme\dist\assets

Not sure why yet but its returning 2 distinct paths there.

[Skrypt]

Ok, I think I found the issue.
It is happening when a "source" contains a "**" in it's path in the Assets.json file. Will take a look.

[gvkries]

Ok, I think I found the issue. It is happening when a "source" contains a "**" in it's path in the Assets.json file. Will take a look.

Thank you 🤗

[Skrypt]

Can you try something on your side?

import fs from "fs-extra";
import { glob } from "glob";
import JSON5 from "json5";
import chalk from "chalk";
import path from "path";

let action = process.argv[2];
const config = JSON5.parse(Buffer.from(process.argv[3], "base64").toString("utf-8"));

let dest = config.dest;
let fileExtension = config.source.split(".").pop();

if (config.dest == undefined) {
    if (config.tags.includes("js") || fileExtension == "js") {
        dest = config.basePath + "/wwwroot/Scripts/";
    } else if (config.tags.includes("css") || fileExtension == "css") {
        dest = config.basePath + "/wwwroot/Styles/";
    }
}

if (config.dryRun) {
    action = "dry-run";
}

// console.log(`copy ${action}`, config);

glob(config.source).then((files) => {
    if (files.length == 0) {
        console.log(chalk.yellow("No files to copy", config.source));
        return;
    }

    const destExists = fs.existsSync(dest);

    if (destExists) {
        const stats = fs.lstatSync(dest);
        if (!stats.isDirectory()) {
            console.log(chalk.red("Destination is not a directory"));
            console.log("Files:", files);
            console.log("Destination:", dest);
            return;
        }
        console.log(chalk.yellow(`Destination ${dest} already exists, files may be overwritten`));
    }

    let baseFolder;

    if (config.source.indexOf("**") > 0) {
        baseFolder = config.source.substring(0, config.source.indexOf("**"));
    }

    files.forEach(async (file) => {
        file = file.replace(/\\/g, "/");
        let relativePath;
        if (baseFolder) {
            relativePath = file.replace(baseFolder, "");
        } else {
            relativePath = path.basename(file);
        }

        const target = path.join(dest, relativePath);

        if (action === "dry-run") {
            console.log(`Dry run (${chalk.gray("from")}, ${chalk.cyan("to")})`, chalk.gray(file), chalk.cyan(target));
        } else {
            await fs.stat(file).then(async (stat) => {
                if (!stat.isDirectory()) {
                    await fs.copy(file, target)
                        .then(() => console.log(`Copied (${chalk.gray("from")}, ${chalk.cyan("to")})`, chalk.gray(file), chalk.cyan(target)))
                        .catch((err) => {
                            console.log(`${chalk.red("Error copying")} (${chalk.gray("from")}, ${chalk.cyan("to")})`, chalk.gray(file), chalk.cyan(target), chalk.red(err));
                            throw err;
                        });
                }
            });
        }
    });
});

Replace the copy.mjs with this code. And let me know if that fixes the issue.

[gvkries]

There are no more errors on 'copy', but the themes are still failing:

theme-theagencytheme exited with code 1
theme-thecomingsoontheme exited with code 1
theme-theblogtheme exited with code 1
Errors occurred!

I don't know why, I only find lots of 'sass' errors in the output. E.g.

[theme-theblogtheme] sass.Exception [Error]: Can't find stylesheet to import.
[theme-theblogtheme]   ╷
[theme-theblogtheme] 7 │ @import "../../../node_modules/bootstrap/scss/functions";
[theme-theblogtheme]   │         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

But I don't know if that's the issue now.

[gvkries]

Here is the full log, if that helps...
yarn log.txt

[Skrypt]

You are supposed to have a node_modules folder that contains a bootstrap folder right there in that theme. If it's not there then you need to run yarn again on OC root folder.

[gvkries]

Oh, sorry. I was playing around and I deleted those, but I forget about it. My fault, it is working now with your fixes to the copy script.

Thanks again for your massive work and for helping me out here too!

[gvkries]

One more (hopefully last) question: If I build from the latest main, it creates the following changes:

Is that expected?

[Skrypt]

If you are staging these changes does it still shows these files with changes?
If so, could you send me a screenshot of one of the diff on these files?

[Skrypt]

@agriffard had the same issue which I did not find anything about yet. I need to understand why Webpack moves that licence into a different spot in that file. My PC and the CI consistently outputs the same for me so far. How can I repro this one is the question.

[gvkries]

Yup, still showing as modified if staged.