feat: buildtime version check for os-dependentLibraries (#981)

### Summary
- Fixes a case making it possible to have different os lib versions in
requirements.txt and globalConfig.json. Added log entry on version
mismatch, example:
```
INFO: Installing os-dependentLibraries.
INFO: Executing: python3 -m pip show --version wrong_lib_name | grep "Version: 1.2.3"
ERROR: Command (python3 -m pip show --version wrong_lib_name | grep "Version: 1.2.3") returned True status code
ERROR: 
OS dependent library wrong_lib_name = 1.2.3 SHOULD be defined in requirements.txt.
When the os dependent library is installed without its dependencies it has to be listed in requirements.txt.
Possible solutions, either:
1. os-dependentLibraries.name[wrong_lib_name].dependencies = True
2. Add wrong_lib_name=1.2.3 in requirements.txt

```
- Cleanup the installed os libs from the ucc_lib_target, making
deployment package smaller.
- `splunktaucclib` presence is checked via `pip`. Check is performed
after
`splunk_add_on_ucc_framework.install_python_libraries.install_libraries`
is called.
Depends on env`PYTHONPATH` to be set up to TA installation target
(example:
`/Users/dkvashnin/splunk/addonfactory-ucc-generator/output/Splunk_TA_UCCExample/lib`)
as `pip show` doesn't accept `--target` option like `pip install` does.

docs/advanced/os-dependent_libraries.md

@@ -2,15 +2,15 @@ This feature allows us to download and unpack libraries with appropriate binarie
 To do this, you need to expand the **meta** section in global config with the **os-dependentLibraries** field. This field takes the following attributes:
 
 
-| Property                                               | Type    | Description                                                                                                                                                                                                                                                              | default value |
-|--------------------------------------------------------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
-| name<span class="required-asterisk">*</span>           | string  | Name of the library we want to download.                                                                                                                                                                                                                                 | -             |
-| version<span class="required-asterisk">*</span>        | string  | Specific version of given library.                                                                                                                                                                                                                                       | -             |
-| dependencies                                           | boolean | Optional parameter which determines whether the "--no-deps" flag will be used. When the value is set to true, the flag "--no-deps" is missing and specified library will be downloaded along with all dependencies. In this case dependency versions are handled by pip. | false         |
-| platform<span class="required-asterisk">*</span>       | string  | The platform for which we want to download the specified library. The value depends on the available wheels for a given library e.g. for this wheel **cryptography-41.0.5-cp37-abi3-manylinux_2_28_x86_64.whl** platform is **manylinux_2_28_x86_64**.                   | -             |
-| python_version<span class="required-asterisk">*</span> | string  | Python version compatible with the library.                                                                                                                                                                                                                              | -             |
-| target<span class="required-asterisk">*</span>         | string  | Path where the selected library will be unpacked.                                                                                                                                                                                                                        | -             |
-| os<span class="required-asterisk">*</span>             | string  | The name of the operating system for which the library is intended. Using this parameter, an appropriate insert into sys.path will be created. It takes 3 values **windows**, **linux** and **darwin**.                                                                  | -             |
+| Property                                               | Type    | Description                                                                                                                                                                                                                                                                                                                        | default value |
+|--------------------------------------------------------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| name<span class="required-asterisk">*</span>           | string  | Name of the library we want to download.                                                                                                                                                                                                                                                                                           | -             |
+| version<span class="required-asterisk">*</span>        | string  | Specific version of given library.                                                                                                                                                                                                                                                                                                 | -             |
+| dependencies                                           | boolean | (Optional) Parameter which determines whether the `--no-deps` flag will be used when installing package from `pip`. When the value is set to `true` the library will be installed along with all its dependencies. When the value is set to `false` *(default)* `{name}={version}` must be present in packages `requirements.txt`. | false         |
+| platform<span class="required-asterisk">*</span>       | string  | The platform for which we want to download the specified library. The value depends on the available wheels for a given library e.g. for this wheel **cryptography-41.0.5-cp37-abi3-manylinux_2_28_x86_64.whl** platform is **manylinux_2_28_x86_64**.                                                                             | -             |
+| python_version<span class="required-asterisk">*</span> | string  | Python version compatible with the library.                                                                                                                                                                                                                                                                                        | -             |
+| target<span class="required-asterisk">*</span>         | string  | Path where the selected library will be unpacked.                                                                                                                                                                                                                                                                                  | -             |
+| os<span class="required-asterisk">*</span>             | string  | The name of the operating system for which the library is intended. Using this parameter, an appropriate insert into sys.path will be created. It takes 3 values **windows**, **linux** and **darwin**.                                                                                                                            | -             |
 
 ### Usage
 

splunk_add_on_ucc_framework/commands/build.py

@@ -686,9 +686,15 @@ def generate(
         os.path.abspath(os.path.join(source, os.pardir, "additional_packaging.py"))
     ):
         sys.path.insert(0, os.path.abspath(os.path.join(source, os.pardir)))
-        from additional_packaging import additional_packaging
+        try:
+            from additional_packaging import additional_packaging
 
-        additional_packaging(ta_name)
+            additional_packaging(ta_name)
+        except ImportError as e:
+            logger.exception(
+                "additional_packaging.py is present but not importable.", e
+            )
+            raise e
 
     if global_config:
         logger.info("Generating OpenAPI file")

splunk_add_on_ucc_framework/install_python_libraries.py

@@ -20,7 +20,7 @@
 import subprocess
 import sys
 from pathlib import Path
-from typing import Sequence, List, Optional
+from typing import List, Optional, Set, Iterable, Dict
 from splunk_add_on_ucc_framework.global_config import OSDependentLibraryConfig
 
 logger = logging.getLogger("ucc_gen")
@@ -34,19 +34,51 @@ class CouldNotInstallRequirements(Exception):
     pass
 
 
-def _subprocess_call(command: str, command_desc: str) -> None:
+def _subprocess_call(
+    command: str,
+    command_desc: Optional[str] = None,
+    env: Optional[Dict[str, str]] = None,
+) -> int:
+    command_desc = command_desc or command
     try:
-        return_code = subprocess.call(command, shell=True)
+        logger.info(f"Executing: {command}")
+        return_code = subprocess.call(command, shell=True, env=env)
         if return_code < 0:
             logger.error(
                 f"Child ({command_desc}) was terminated by signal {-return_code}"
             )
-            raise CouldNotInstallRequirements
         if return_code > 0:
             logger.error(f"Command ({command_desc}) returned {return_code} status code")
-            raise CouldNotInstallRequirements
+        return return_code
     except OSError as e:
         logger.error(f"Execution ({command_desc}) failed due to {e}")
+        raise e
+
+
+def _pip_install(installer: str, command: str, command_desc: str) -> None:
+    cmd = f"{installer} -m pip install {command}"
+    try:
+        return_code = _subprocess_call(command=cmd, command_desc=command_desc)
+        if return_code != 0:
+            raise CouldNotInstallRequirements
+    except OSError as e:
+        raise CouldNotInstallRequirements from e
+
+
+def _pip_is_lib_installed(
+    installer: str, target: str, libname: str, version: Optional[str] = None
+) -> bool:
+    lib_installed_cmd = f"{installer} -m pip show --version {libname}"
+    lib_version_match_cmd = f'{lib_installed_cmd} | grep "Version: {version}"'
+
+    cmd = lib_version_match_cmd if version else lib_installed_cmd
+
+    try:
+        my_env = os.environ.copy()
+        my_env["PYTHONPATH"] = target
+        return_code = _subprocess_call(command=cmd, env=my_env)
+        return return_code == 0
+    except OSError as e:
         raise CouldNotInstallRequirements from e
 
 
@@ -69,38 +101,41 @@ def install_python_libraries(
     path_to_requirements_file = os.path.join(source_path, "lib", "requirements.txt")
     if os.path.isfile(path_to_requirements_file):
         logger.info(f"Installing requirements from {path_to_requirements_file}")
-        if includes_ui:
-            ucc_library_present = _check_ucc_library_in_requirements_file(
-                path_to_requirements_file
-            )
-            if not ucc_library_present:
-                raise SplunktaucclibNotFound(
-                    f"splunktaucclib is not found in {path_to_requirements_file}. "
-                    f"Please add it there because this add-on has UI."
-                )
         if not os.path.exists(ucc_lib_target):
             os.makedirs(ucc_lib_target)
         install_libraries(
-            path_to_requirements_file,
-            ucc_lib_target,
-            python_binary_name,
+            requirements_file_path=path_to_requirements_file,
+            installation_path=ucc_lib_target,
+            installer=python_binary_name,
         )
-
-        if os_libraries:
-            install_os_dependent_libraries(
-                ucc_lib_target, python_binary_name, os_libraries
+        if includes_ui and not _pip_is_lib_installed(
+            installer=python_binary_name,
+            target=ucc_lib_target,
+            libname="splunktaucclib",
+        ):
+            raise SplunktaucclibNotFound(
+                f"splunktaucclib is not found in {path_to_requirements_file}. "
+                f"Please add it there because this add-on has UI."
             )
 
-        packages_to_remove = (
+        cleanup_libraries = install_os_dependent_libraries(
+            ucc_lib_target=ucc_lib_target,
+            installer=python_binary_name,
+            os_libraries=os_libraries,
+        )
+
+        packages_to_remove = {
             "setuptools",
             "bin",
             "pip",
             "distribute",
             "wheel",
-        )
-        remove_package_from_installed_path(
-            ucc_lib_target,
-            packages_to_remove,
+        }
+        # we can remove os-dependent libraries from the installation_path to save some space.
+        packages_to_remove.update(cleanup_libraries)
+        remove_packages(
+            installation_path=ucc_lib_target,
+            package_names=packages_to_remove,
         )
 
         remove_execute_bit(ucc_lib_target)
@@ -121,11 +156,8 @@ def install_libraries(
     """
 
     pip_version = "23.1.2"
-    pip_update_command = f"{installer} -m pip install --upgrade pip=={pip_version}"
+    pip_update_command = f"--upgrade pip=={pip_version}"
     pip_install_command = (
-        f"{installer} "
-        f"-m pip "
-        f"install "
         f'-r "{requirements_file_path}" '
         f"--no-compile "
         f"--prefer-binary "
@@ -134,13 +166,15 @@ def install_libraries(
         f'--target "{installation_path}"'
     )
 
-    _subprocess_call(pip_update_command, "pip upgrade")
-    _subprocess_call(pip_install_command, "pip install")
+    _pip_install(
+        installer=installer, command=pip_update_command, command_desc="pip upgrade"
+    )
+    _pip_install(
+        installer=installer, command=pip_install_command, command_desc="pip install"
+    )
 
 
-def remove_package_from_installed_path(
-    installation_path: str, package_names: Sequence[str]
-) -> None:
+def remove_packages(installation_path: str, package_names: Iterable[str]) -> None:
     p = Path(installation_path)
     for package_name in package_names:
         for o in p.glob(f"{package_name}*"):
@@ -164,31 +198,58 @@ def remove_execute_bit(installation_path: str) -> None:
 
 
 def install_os_dependent_libraries(
-    ucc_lib_target: str, installer: str, os_libraries: List[OSDependentLibraryConfig]
-) -> None:
-    for package in os_libraries:
-        target_path = os.path.join(ucc_lib_target, os.path.normpath(package.target))
+    ucc_lib_target: str,
+    installer: str,
+    os_libraries: Optional[List[OSDependentLibraryConfig]],
+) -> Set[str]:
+    cleanup_libraries: Set[str] = set()
+
+    if not os_libraries:
+        logger.info("No os-dependentLibraries to install.")
+        return cleanup_libraries
+
+    logger.info("Installing os-dependentLibraries.")
+    for os_lib in os_libraries:
+        if os_lib.dependencies is False and not _pip_is_lib_installed(
+            installer=installer,
+            target=ucc_lib_target,
+            libname=os_lib.name,
+            version=os_lib.version,
+        ):
+            logger.error(
+                f"""
+OS dependent library {os_lib.name} = {os_lib.version} SHOULD be defined in requirements.txt.
+When the os dependent library is installed without its dependencies it has to be listed in requirements.txt.
+Possible solutions, either:
+1. os-dependentLibraries.name[{os_lib.name}].dependencies = True
+2. Add {os_lib.name}={os_lib.version} in requirements.txt
+"""
+            )
+            raise CouldNotInstallRequirements
 
+        target_path = os.path.join(ucc_lib_target, os.path.normpath(os_lib.target))
         if not os.path.exists(target_path):
             os.makedirs(target_path)
 
         pip_download_command = (
-            f"{installer} "
-            f"-m pip "
-            f"install "
-            f"{package.deps_flag} "
-            f"--platform {package.platform} "
-            f"--python-version {package.python_version} "
+            f"{os_lib.deps_flag} "
+            f"--platform {os_lib.platform} "
+            f"--python-version {os_lib.python_version} "
             f"--target {target_path}"
             f" --only-binary=:all: "
-            f"{package.name}=={package.version}"
+            f"{os_lib.name}=={os_lib.version}"
         )
 
-        logger.info(f"Executing: {pip_download_command}")
         try:
-            _subprocess_call(pip_download_command, "pip download")
+            _pip_install(
+                installer=installer,
+                command=pip_download_command,
+                command_desc="pip download",
+            )
         except CouldNotInstallRequirements:
             logger.error(
                 "Downloading process failed. Please verify parameters in the globalConfig.json file."
             )
             sys.exit("Package building process interrupted.")
+        cleanup_libraries.add(os_lib.name)
+    return cleanup_libraries

tests/testdata/test_addons/package_global_config_everything/globalConfig.json

@@ -1354,7 +1354,7 @@
     "meta": {
         "name": "Splunk_TA_UCCExample",
         "restRoot": "splunk_ta_uccexample",
-        "version": "5.34.1R3fad1a25",
+        "version": "5.35.1Rb83f2c0b",
         "displayName": "Splunk UCC test Add-on",
         "schemaVersion": "0.0.3"
     }