Configurable linter commands (#4482)

* Configurable linter commands * [MegaLinter] Apply linters fixes * Change parameter name from parallel to run_before_linters * Fix PR comments * [MegaLinter] Apply linters fixes * Add run_after_linters parameter * Rename linter parameters * [MegaLinter] Apply linters fixes --------- Co-authored-by: bdovaz <bdovaz@users.noreply.github.com> Co-authored-by: bdovaz <950602+bdovaz@users.noreply.github.com>
oxsecurity · Jan 19, 2025 · 995935c · 995935c
1 parent dcbf7f7
commit 995935c
Show file tree

Hide file tree

Showing 6 changed files with 78 additions and 21 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -11,6 +11,7 @@ Note: Can be used with `oxsecurity/megalinter@beta` in your GitHub Action mega-l
 - Core
   - PHP Linters use now the `bartlett/sarif-php-converters` first official release 1.0.0 to generate SARIF reports
   - [Upgrade PHP engine from 8.3 to 8.4](https://github.com/oxsecurity/megalinter/issues/4351) and allow Psalm 5.26 to run on this context (by @llaville)
+  - Linters can specify in the pre/post commands with a `run_before_linters` / `run_after_linters` parameter whether the command is to be executed before/after the execution of the linters themselves (by @bdovaz in [#4482](https://github.com/oxsecurity/megalinter/pull/4482))
 
 - New linters
   - Reactivate clj-style (Clojure formatter) since its bug is fixed

diff --git a/README.md b/README.md
@@ -77,8 +77,8 @@ Before going below, see [**Online Documentation Web Site which has a much easier
   - [Which version to use ?](#which-version-to-use-)
   - [GitHub Action](#github-action)
   - [GitLab CI](#gitlab-ci)
-    - [Manual Setup](#manual-setup)
-    - [Using R2Devops](#using-r2devops)
+- [Manual Setup](#manual-setup)
+  - [Using R2Devops](#using-r2devops)
   - [Azure Pipelines](#azure-pipelines)
     - [Single Repository](#single-repository)
     - [Central Repository](#central-repository)
@@ -1182,6 +1182,7 @@ PRE_COMMANDS:
   - command: npm install eslint-plugin-whatever
     cwd: root        # Will be run at the root of MegaLinter docker image
     secured_env: true  # True by default, but if defined to false, no global variable will be hidden (for example if you need GITHUB_TOKEN)
+    run_before_linters: True # Will be run before the execution of the linters themselves, required for npm/pip commands that cannot be run in parallel
   - command: echo "pre-test command has been called"
     cwd: workspace   # Will be run at the root of the workspace (usually your repository root)
     continue_if_failed: False  # Will stop the process if command is failed (return code > 0)
@@ -1193,17 +1194,21 @@ PRE_COMMANDS:
     cwd: workspace   # Will be run at the root of the workspace (usually your repository root)
     continue_if_failed: False  # Will stop the process if command is failed (return code > 0)
     tag: before_plugins # Tag indicating that the command will be run before loading plugins
+  - command: echo "Some command called after running MegaLinter linters"
+    run_after_linters: True # Will be run after the execution of the linters themselves
 ```
 
-| Property               | Description                                                                                                                              | Default value |
-|------------------------|------------------------------------------------------------------------------------------------------------------------------------------|---------------|
-| **command**            | Command line to run                                                                                                                      | Mandatory     |
-| **cwd**                | Directory where to run the command (`workspace` or `root`)                                                                               | `workspace`   |
-| **secured_env**        | Apply filtering of secured env variables before calling the command (default true)<br/>Be careful if you disable it !                    | `true`        |
-| **continue_if_failed** | If set to `false`, stop MegaLinter process in case of command failure                                                                    | `true`        |
-| **venv**               | If set, runs the command into the related python venv                                                                                    | <!-- -->      |
-| **output_variables**   | ENV variables to get from output after running the commands, and store in MegaLinter ENV context, so they can be reused in next commands | `[]`          |
-| **tag**                | Tag defining at which commands entry point the command will be run   (available tags: `before_plugins`)                                  | <!-- -->      |
+| Property               | Description                                                                                                                                     | Default value |
+|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| **command**            | Command line to run                                                                                                                             | Mandatory     |
+| **cwd**                | Directory where to run the command (`workspace` or `root`)                                                                                      | `workspace`   |
+| **run_before_linters** | If set to `true`, runs the command before the execution of the linters themselves, required for npm/pip commands that cannot be run in parallel | `false`       |
+| **run_after_linters**  | If set to `true`, runs the command after the execution of the linters themselves                                                                | `false`       |
+| **secured_env**        | Apply filtering of secured env variables before calling the command (default true)<br/>Be careful if you disable it !                           | `true`        |
+| **continue_if_failed** | If set to `false`, stop MegaLinter process in case of command failure                                                                           | `true`        |
+| **venv**               | If set, runs the command into the related python venv                                                                                           | <!-- -->      |
+| **output_variables**   | ENV variables to get from output after running the commands, and store in MegaLinter ENV context, so they can be reused in next commands        | `[]`          |
+| **tag**                | Tag defining at which commands entry point the command will be run   (available tags: `before_plugins`)                                         | <!-- -->      |
 
 <!-- config-precommands-section-end -->
 <!-- config-postcommands-section-start -->

diff --git a/megalinter/Linter.py b/megalinter/Linter.py
@@ -805,7 +805,7 @@ def update_active_if_file_found(self):
             self.active_only_if_file_found.append(self.config_file_name)
 
     # Processes the linter
-    def run(self):
+    def run(self, run_commands_before_linters=None, run_commands_after_linters=None):
         self.start_perf = perf_counter()
 
         # Initialize linter reports
@@ -816,7 +816,9 @@ def run(self):
         self.before_lint_files()
 
         # Run commands defined in descriptor, or overridden by user in configuration
-        pre_post_factory.run_linter_pre_commands(self.master, self)
+        pre_post_factory.run_linter_pre_commands(
+            self.master, self, run_commands_before_linters
+        )
 
         # Lint each file one by one
         if self.cli_lint_mode == "file":
@@ -871,7 +873,9 @@ def run(self):
             os.remove(self.remote_ignore_file_to_delete)
 
         # Run commands defined in descriptor, or overridden by user in configuration
-        pre_post_factory.run_linter_post_commands(self.master, self)
+        pre_post_factory.run_linter_post_commands(
+            self.master, self, run_commands_after_linters
+        )
 
         # Generate linter reports
         self.elapsed_time_s = perf_counter() - self.start_perf

diff --git a/megalinter/MegaLinter.py b/megalinter/MegaLinter.py
@@ -48,7 +48,7 @@ def run_linters(linters, request_id):
     global REQUEST_CONFIG
     config.set_config(request_id, REQUEST_CONFIG)
     for linter in linters:
-        linter.run()
+        linter.run(run_commands_before_linters=False, run_commands_after_linters=False)
     return linters
 
 
@@ -244,7 +244,17 @@ def run(self):
             config.get(self.request_id, "PARALLEL", "true") == "true"
             and len(active_linters) > 1
         ):
+            for active_linter in active_linters:
+                pre_post_factory.run_linter_pre_commands(
+                    active_linter.master, active_linter, run_before_linters=True
+                )
+
             self.process_linters_parallel(active_linters, linters_do_fixes)
+
+            for active_linter in active_linters:
+                pre_post_factory.run_linter_post_commands(
+                    active_linter.master, active_linter, run_after_linters=True
+                )
         else:
             self.process_linters_serial(active_linters)
 

diff --git a/megalinter/descriptors/schemas/megalinter-descriptor.jsonschema.json b/megalinter/descriptors/schemas/megalinter-descriptor.jsonschema.json
@@ -37,6 +37,25 @@
       "title": "Command",
       "type": "object"
     },
+    "linter_command_info": {
+      "description": "Command information",
+      "$ref": "#/definitions/command_info",
+      "properties": {
+        "run_before_linters": {
+          "default": false,
+          "title": "Process command before running linters",
+          "type": "boolean"
+        },
+        "run_after_linters": {
+          "default": false,
+          "title": "Process command after running linters",
+          "type": "boolean"
+        }
+      },
+      "required": [],
+      "title": "Command",
+      "type": "object"
+    },
     "enum_flavors": {
       "enum": [
         "all_flavors",
@@ -1232,7 +1251,7 @@
               ]
             ],
             "items": {
-              "$ref": "#/definitions/command_info"
+              "$ref": "#/definitions/linter_command_info"
             },
             "title": "Linter Pre-run commands",
             "type": "array"
@@ -1255,7 +1274,7 @@
               ]
             ],
             "items": {
-              "$ref": "#/definitions/command_info"
+              "$ref": "#/definitions/linter_command_info"
             },
             "title": "Linter Pre-run commands",
             "type": "array"

diff --git a/megalinter/pre_post_factory.py b/megalinter/pre_post_factory.py
@@ -34,19 +34,37 @@ def run_descriptor_post_commands(mega_linter, descriptor_id):
 
 
 # Commands to run before a linter (defined in descriptors)
-def run_linter_pre_commands(mega_linter, linter):
+def run_linter_pre_commands(mega_linter, linter, run_before_linters=None):
     if linter.pre_commands is not None:
+        filtered_commands: list = []
+
+        if run_before_linters is None:
+            filtered_commands = linter.pre_commands
+        else:
+            for command_info in linter.pre_commands:
+                if command_info.get("run_before_linters", False) is run_before_linters:
+                    filtered_commands += command_info
+
         return run_commands(
-            linter.pre_commands, "[Pre][" + linter.name + "]", mega_linter, linter
+            filtered_commands, "[Pre][" + linter.name + "]", mega_linter, linter
         )
     return []
 
 
 # Commands to run before a linter (defined in descriptors)
-def run_linter_post_commands(mega_linter, linter):
+def run_linter_post_commands(mega_linter, linter, run_after_linters=None):
     if linter.post_commands is not None:
+        filtered_commands: list = []
+
+        if run_after_linters is None:
+            filtered_commands = linter.post_commands
+        else:
+            for command_info in linter.post_commands:
+                if command_info.get("run_after_linters", False) is run_after_linters:
+                    filtered_commands += command_info
+
         return run_commands(
-            linter.post_commands, "[Post][" + linter.name + "]", mega_linter, linter
+            filtered_commands, "[Post][" + linter.name + "]", mega_linter, linter
         )
     return []