feat: release v5.25.0 (#733)

* refactor: use subprocess.call instead of os.system (#718)

Docs: https://docs.python.org/3/library/subprocess.html#replacing-os-system

* chore: warn that running just ucc-gen will be deprecated (#719)

* refactor: remove time from logs (#722)

* chore: add a script to compare output from 2 different versions of add-on (#723)

* fix: change the order in the sys.path in import_declare_test (#724)

The add-on should use libraries packaged with it first and only
then try to use libraries provided by Splunk.

* docs: initial version of describing solnlib and splunktaucclib (#726)

* fix: openapi.json should not contain disabled property for input creation (#727)

* fix: openapi.json should not contain disabled property for input creation ; resolves #725

* fix: help text for generated account (#728)

* chore: don't show traceback when not a git repo (#734)

Instead of showing a traceback, show an error message in logs and
suggest users a solution.

* test: mimic code structure in unit tests (#735)

* chore: removed license from test files

* test: mimic code structure in unit tests

---------

Co-authored-by: hsekowski-splunk <74253926+hsekowski-splunk@users.noreply.github.com>
Co-authored-by: Daniel Federschmidt <daniel@federschmidt.xyz>

.licenserc.yaml

@@ -37,3 +37,4 @@ header:
     - "splunk_add_on_ucc_framework/commands/init_template/**"
     - "splunk_add_on_ucc_framework/commands/import_from_aob.sh"
     - "splunk_add_on_ucc_framework/commands/imports.py"
+    - "scripts/compare_different_ucc_versions_output.sh"

docs/index.md

@@ -1,14 +1,10 @@
 # Overview
 
-`splunk-add-on-ucc-framework` is a framework to generate UI based Splunk
-Add-ons. It includes UI, Rest handler, Modular input, Oauth, Alert
+`splunk-add-on-ucc-framework` is a framework to generate UI-based Splunk
+Add-ons. It includes UI, REST handlers, Modular inputs, OAuth and Alert
 action templates.
 
-> After UCC 5.2 Python 2 specific libraries are not supported anymore.
-> This means if the add-on has package/lib/py2/requirements.txt they
-> will not be installed while running ucc-gen command. Therefore,
-> modular inputs that are supposed to run on Python 2 will not be
-> supported by UCC.
+Only add-ons that use Python 3 are supported.
 
 Available as a GitHub action here:
 <https://github.com/splunk/addonfactory-ucc-generator-action>
@@ -21,18 +17,24 @@ as well.
 UCC stands for Universal Configuration Console. The purpose of having a
 framework for add-on generation is to simplify the process of add-on
 creation for the developers. UCC 5 uses [SplunkUI](https://splunkui.splunk.com/) 
-which is a new UI framework based on React.
+which is a new UI framework based on React. UCC UI repository can be found
+[here](https://github.com/splunk/addonfactory-ucc-base-ui).
+
+UCC-based add-ons are being powered by another Splunk libraries:
+[`solnlib`](https://github.com/splunk/addonfactory-solutions-library-python) and
+[`splunktaucclib`](https://github.com/splunk/addonfactory-ucc-library). More
+information [here](ucc_related_libraries.md).
 
 ## Features
 
 * Generate UI (`appserver` folder)
 * Generate Python REST handlers to support UI CRUD operations (`bin` folder)
-* Generate OpenAPI description document (`static/openapi.json` file)
+* Generate OpenAPI description document (`static/openapi.json` file) (more info [here](openapi.md))
 * Generate `.conf` files (more info [here](dot_conf_files.md))
 * Install Python requirements (`lib` folder)
 * Generate metadata files (`metadata` folder)
-* Possibility to extend UI with custom code
-* Possibility to extend the build process via `additional_packaging.py` file
+* Possibility to extend UI with custom code (more info [here](custom_ui_extensions/custom_hook.md))
+* Possibility to extend the build process via `additional_packaging.py` file (more info [here](additional_packaging.md))
 
 ## Installation
 

docs/troubleshooting.md

@@ -30,3 +30,14 @@ It can be because of:
 
 * there is no such library in the `package/lib` folder -> you need to check your `package/lib/requirements.txt` file to make sure that you have it as part of the requirements
 * there is no `import import_declare_test` at the top of your modular input file -> add the import mentioned to the top of your modular input file
+
+## Compare add-ons generated by 2 different versions of the `ucc-gen build`
+
+`scripts` folder has a script `compare_different_ucc_versions_output.sh` which
+can be used to compare the output of the add-on built by 2 different versions of
+the `ucc-gen build` command.
+
+To be able to run it, copy the script to the add-on folder, read the
+instructions in the script itself. If it does not work for your particular case,
+feel free to adjust it yourself or file a feature request for us to improve
+something.

docs/ucc_related_libraries.md

@@ -0,0 +1,34 @@
+# UCC-related libraries
+
+There are two UCC-related libraries:
+
+* [`solnlib`](https://github.com/splunk/addonfactory-solutions-library-python)
+* [`splunktaucclib`](https://github.com/splunk/addonfactory-ucc-library)
+
+## `solnlib`
+
+`solnlib` contains a number of functions and classes that can be used during
+the add-on development. The documentation can be found
+[here](https://splunk.github.io/addonfactory-solutions-library-python).
+
+Commonly used modules are:
+
+* `log` - file-based logging to `$SPLUNK_HOME/var/log/splunk` folder
+* `modular_input.checkpointers` - contains classes to manage checkpoints
+
+Below is the example of the code that can be used to get a logger for a
+specific add-on.
+
+```python
+import logging
+
+from solnlib import log
+
+
+def logger_for_input(input_name: str) -> logging.Logger:
+    return log.Logs().get_logger(f"<add-on name>_{input_name}")
+```
+
+## `splunktaucclib`
+
+`splunktaucclib` powers backend of the add-on.

example/globalConfig.json

@@ -161,7 +161,7 @@
     "meta": {
         "name": "Splunk_TA_dummy_data",
         "restRoot": "Splunk_TA_dummy_data",
-        "version": "5.23.2R593316cd",
+        "version": "5.24.0R53833250",
         "displayName": "Splunk_TA_dummy_data",
         "schemaVersion": "0.0.3"
     }

mkdocs.yml

@@ -46,6 +46,7 @@ nav:
   - Table: "table.md"
   - Additional packaging: "additional_packaging.md"
   - OpenAPI: "openapi.md"
+  - UCC-related libraries: "ucc_related_libraries.md"
   - Custom UI Extensions:
     - Custom Hook: "custom_ui_extensions/custom_hook.md"
     - Custom Control: "custom_ui_extensions/custom_control.md"

scripts/compare_different_ucc_versions_output.sh

@@ -0,0 +1,40 @@
+#! /usr/bin/env bash
+# The script installs 2 different version of the `splunk-add-on-ucc-framework`,
+# builds the add-on, cleans up some folders and then prints the difference between
+# the 2 built versions.
+#
+# The script assumes uses `python3` binary (you can change it for your needs) and needs
+# 2 versions of the `splunk-add-on-ucc-framework` as parameters.
+# An example of calling the script:
+# `./compare_ucc_versions 5.19.0 5.24.0`
+
+set -euo pipefail
+
+python3 -m venv .venv
+source .venv/bin/activate
+
+rm -rf output_$1
+rm -rf output_$2
+
+echo "Installing splunk-add-on-ucc-framework==$1"
+pip install splunk-add-on-ucc-framework==$1 > /dev/null 2>&1
+.venv/bin/ucc-gen > /dev/null 2>&1
+mv output/ output_$1/
+
+echo "Installing splunk-add-on-ucc-framework==$2"
+pip install splunk-add-on-ucc-framework==$2 > /dev/null 2>&1
+.venv/bin/ucc-gen > /dev/null 2>&1
+mv output/ output_$2/
+
+echo "Removing lib/ and appserver/static/jb/"
+rm -rf output_$1/*/lib/
+rm -rf output_$2/*/lib/
+rm -rf output_$1/*/appserver/static/js/
+rm -rf output_$2/*/appserver/static/js/
+
+diff -bur output_$1 output_$2
+
+rm -rf output_$1
+rm -rf output_$2
+
+deactivate

splunk_add_on_ucc_framework/__init__.py

@@ -19,7 +19,7 @@
 
 logger = logging.getLogger("ucc_gen")
 logger.setLevel(logging.INFO)
-formatter = logging.Formatter("%(asctime)s %(levelname)s: %(message)s")
+formatter = logging.Formatter("%(levelname)s: %(message)s")
 stream_handler = logging.StreamHandler()
 stream_handler.setLevel(logging.INFO)
 stream_handler.setFormatter(formatter)

splunk_add_on_ucc_framework/commands/build.py

@@ -417,6 +417,13 @@ def _get_addon_version(addon_version: Optional[str]) -> str:
     if not addon_version:
         try:
             return utils.get_version_from_git()
+        except exceptions.IsNotAGitRepo:
+            logger.error(
+                "Could not determine a version using Git (maybe not a Git "
+                "repository?). Use `--ta-version` to specify the version you "
+                "want."
+            )
+            exit(1)
         except exceptions.CouldNotVersionFromGitException:
             logger.error(
                 "Could not find the proper version from git tags. "

splunk_add_on_ucc_framework/commands/init_template/{{cookiecutter.addon_name}}/globalConfig.json

@@ -24,7 +24,7 @@
                             "validators": [
                                 {
                                     "type": "regex",
-                                    "errorMsg": "Input Name must begin with a letter and consist exclusively of alphanumeric characters and underscores.",
+                                    "errorMsg": "Account Name must begin with a letter and consist exclusively of alphanumeric characters and underscores.",
                                     "pattern": "^[a-zA-Z]\\w*$"
                                 },
                                 {
@@ -35,7 +35,7 @@
                                 }
                             ],
                             "field": "name",
-                            "help": "A unique name for the data input.",
+                            "help": "A unique name for the account.",
                             "required": true
                         },
                         {

splunk_add_on_ucc_framework/commands/openapi_generator/ucc_to_oas.py

@@ -124,27 +124,13 @@ def __add_schemas_object(
             open_api_object.components.schemas[tab.name] = __get_schema_object(
                 name=tab.name, entities=tab.entity
             )
-        additional_input_entities = [
-            json_to_object.DataClasses(
-                json={
-                    "field": "disabled",
-                    "type": "singleSelect",
-                    "options": {
-                        "autoCompleteFields": [
-                            {"value": "0"},
-                            {"value": "1"},
-                        ]
-                    },
-                }
-            )
-        ]
         if hasattr(global_config.pages, "inputs") and hasattr(
             global_config.pages.inputs, "services"
         ):
             for service in global_config.pages.inputs.services:
                 open_api_object.components.schemas[service.name] = __get_schema_object(
                     name=service.name,
-                    entities=service.entity + additional_input_entities,
+                    entities=service.entity,
                 )
     return open_api_object
 

splunk_add_on_ucc_framework/commands/rest_builder/global_config_post_processor.py

@@ -36,7 +36,7 @@ class GlobalConfigPostProcessor:
 ta_name = '{ta_name}'
 pattern = re.compile(r'[\\\\/]etc[\\\\/]apps[\\\\/][^\\\\/]+[\\\\/]bin[\\\\/]?$')
 new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
-new_paths.append(os.path.join(dirname(dirname(__file__)), "lib"))
+new_paths.insert(0, os.path.join(dirname(dirname(__file__)), "lib"))
 new_paths.insert(0, os.path.sep.join([os.path.dirname(__file__), ta_name]))
 sys.path = new_paths
 """

splunk_add_on_ucc_framework/exceptions.py

@@ -15,3 +15,7 @@
 #
 class CouldNotVersionFromGitException(Exception):
     pass
+
+
+class IsNotAGitRepo(Exception):
+    pass

splunk_add_on_ucc_framework/install_python_libraries.py

@@ -17,6 +17,7 @@
 import os
 import shutil
 import stat
+import subprocess
 from pathlib import Path
 from typing import Sequence
 
@@ -27,6 +28,26 @@ class SplunktaucclibNotFound(Exception):
     pass
 
 
+class CouldNotInstallRequirements(Exception):
+    pass
+
+
+def _subprocess_call(command: str, command_desc: str) -> None:
+    try:
+        return_code = subprocess.call(command, shell=True)
+        if return_code < 0:
+            logger.error(
+                f"Child ({command_desc}) was terminated by signal {-return_code}"
+            )
+            raise CouldNotInstallRequirements
+        if return_code > 0:
+            logger.error(f"Command ({command_desc}) returned {return_code} status code")
+            raise CouldNotInstallRequirements
+    except OSError as e:
+        logger.error(f"Execution ({command_desc}) failed due to {e}")
+        raise CouldNotInstallRequirements from e
+
+
 def _check_ucc_library_in_requirements_file(path_to_requirements: str) -> bool:
     with open(path_to_requirements) as f_reqs:
         content = f_reqs.readlines()
@@ -104,6 +125,24 @@ def install_libraries(
         os.system(installer + " -m pip install pip --upgrade")
         os.system(install_cmd)
 
+    if not os.path.exists(installation_path):
+        os.makedirs(installation_path)
+    pip_install_command = (
+        f"{installer} "
+        f"-m pip "
+        f"install "
+        f'-r "{requirements_file_path}" '
+        f"--no-compile "
+        f"--prefer-binary "
+        f"--ignore-installed "
+        f"--use-deprecated=legacy-resolver "
+        f'--target "{installation_path}"'
+    )
+    pip_update_command = f"{installer} -m pip install pip --upgrade"
+
+    _subprocess_call(pip_update_command, "pip upgrade")
+    _subprocess_call(pip_install_command, "pip install")
+
 
 def remove_package_from_installed_path(
     installation_path: str, package_names: Sequence[str]

splunk_add_on_ucc_framework/main.py

@@ -16,11 +16,14 @@
 import argparse
 import sys
 from typing import Optional, Sequence
+import logging
 
 from splunk_add_on_ucc_framework.commands import build
 from splunk_add_on_ucc_framework.commands import init
 from splunk_add_on_ucc_framework.commands import import_from_aob
 
+logger = logging.getLogger("ucc_gen")
+
 
 # This is a necessary change to have, so we don't release a breaking change.
 # This class adds a default subparser if none specified. In our case it will be
@@ -48,6 +51,10 @@ def _parse_known_args(self, arg_strings, *args, **kwargs):
                 if subparser_found:
                     break
             else:
+                logger.warning(
+                    "Please use `ucc-gen build` if you want to build "
+                    "an add-on, using just `ucc-gen` will be deprecated"
+                )
                 arg_strings = [d_sp] + arg_strings
         return super()._parse_known_args(arg_strings, *args, **kwargs)
 

splunk_add_on_ucc_framework/utils.py

@@ -34,7 +34,10 @@ def dump_yaml_config(config: Dict[Any, Any], file_path: str):
 
 
 def get_version_from_git():
-    version = dunamai.Version.from_git()
+    try:
+        version = dunamai.Version.from_git()
+    except RuntimeError:
+        raise exceptions.IsNotAGitRepo()
     if not version.stage:
         stage = "R"
     else:

tests/__init__.py

@@ -1,16 +0,0 @@
-#
-# Copyright 2021 Splunk Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-#

tests/testdata/expected_addons/expected_addon_after_init/demo_addon_for_splunk/globalConfig.json

@@ -24,7 +24,7 @@
                             "validators": [
                                 {
                                     "type": "regex",
-                                    "errorMsg": "Input Name must begin with a letter and consist exclusively of alphanumeric characters and underscores.",
+                                    "errorMsg": "Account Name must begin with a letter and consist exclusively of alphanumeric characters and underscores.",
                                     "pattern": "^[a-zA-Z]\\w*$"
                                 },
                                 {
@@ -35,7 +35,7 @@
                                 }
                             ],
                             "field": "name",
-                            "help": "A unique name for the data input.",
+                            "help": "A unique name for the account.",
                             "required": true
                         },
                         {

tests/testdata/expected_addons/expected_output_global_config_configuration/Splunk_TA_UCCExample/bin/import_declare_test.py

@@ -7,6 +7,6 @@
 ta_name = 'Splunk_TA_UCCExample'
 pattern = re.compile(r'[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$')
 new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
-new_paths.append(os.path.join(dirname(dirname(__file__)), "lib"))
+new_paths.insert(0, os.path.join(dirname(dirname(__file__)), "lib"))
 new_paths.insert(0, os.path.sep.join([os.path.dirname(__file__), ta_name]))
 sys.path = new_paths

tests/testdata/expected_addons/expected_output_global_config_inputs_configuration_alerts/Splunk_TA_UCCExample/bin/import_declare_test.py

@@ -7,6 +7,6 @@
 ta_name = 'Splunk_TA_UCCExample'
 pattern = re.compile(r'[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$')
 new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]
-new_paths.append(os.path.join(dirname(dirname(__file__)), "lib"))
+new_paths.insert(0, os.path.join(dirname(dirname(__file__)), "lib"))
 new_paths.insert(0, os.path.sep.join([os.path.dirname(__file__), ta_name]))
 sys.path = new_paths