Support profile notion in Jira, Polarion and Superset adapter configs/integrations

.gitignore

@@ -138,6 +138,9 @@ venv.bak/
 # mkdocs documentation
 /site
 
+# vscode workspace
+.vscode/
+
 # mypy
 .mypy_cache/
 .dmypy.json
@@ -169,4 +172,7 @@ cython_debug/
 # Development Scripts
 dev/
 
-temp/
+temp/
+
+# Custom vscode workspace files at the root
+/*.code-workspace

.vscode/templates.code-snippets

@@ -25,7 +25,7 @@
 			"#",
 			"# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"",
 			"# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE",
-			"# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICU5LAR PURPOSE ARE",
+			"# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE",
 			"# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE",
 			"# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL",
 			"# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR",

doc/design/profile_mgmt/profile_mgmt.md

@@ -0,0 +1,40 @@
+# Unified profile management (ProfileMgr component)
+
+See also [ticket reference](https://github.com/NewTec-GmbH/pyMetricCli/issues/16) for some background.
+
+## Current state
+
+### Notion of "profile" in pyJiraCli
+
+Jira login data (server/credentials/cert) can be stored as a profile, which is a directory containing .data JSON file and .cert file.
+Support for user/password storage in the profile shall be added with this feature as well.
+
+### Server/token and user/password config in pyMetricCli(adapter)
+
+pyMetricCli requires an adapter (project-specific ingested python config/handler), which contains direct Jira, Polarion and Superset configuration.
+
+This includes the server URL and the token (update of pyPolarionCli documentation for the latter needed), but doesn't allow for for others like cert.
+Currently pyMetricCli can authenticate at Jira only with token (although pyJiraCli would support user/password through CLI).
+So there are some discrepancies between the tools and the adapter config (which pyMetricCli uses).
+
+See [example adapter](https://github.com/NewTec-GmbH/pyMetricCli/blob/30be6a2e8777e0c7bf99063efef30ef4097a488c/examples/adapter/adapter.py).
+
+## Work items proposal
+
+* Create pyProfileMgr component:
+  * Move profile handling (create/delete/update/list) to pyProfileMgr, which shall be used by pyJiraCli (through existing cmd_profile) and pyMetricCli (read access to profile using ProfileMgr).
+  * Add cmd_profile to pyProfileMgr (similar to current pyJiraCli cmd_profile), to allow create/delete/update/list
+  of profiles with a type (currently Jira, Polarion or Superset) and corresponding uniform data attributes.
+* Introduce "profile" notion in pyMetricCli and the adapter config:
+  * Reference profiles with their name (check type) in the adapter config, having precedence before other config.
+  * Decide what to do with server/token (jira_config), username/password/server/token (polarion_config) and username/password/server (superset_config) in the adapter file:
+    * Alternative 1: Remove it (breaking change), i.e. make profiles on pyMetricCli level mandatory.
+    * Alternative 2: Keep current jira_config/polarion_config/superset_config settings in the adapter file and allow for profile references in addition (which should have precedence).
+* Allow persistence of user/password in the profile data file (mandatory for Superset). There is no strict need to encrypt them as the file is stored locally in the user profile folder.
+
+## Open points
+
+* Decide between alternative 1 and 2, i.e. make profiles mandatory in pyMetricCli (removing other authentication config settings) or additional?
+* Resolve some discrepancies along the way:
+  * pyJiraCli supports user/password authentication through CLI options, but the adapter file only supports token for Jira. The new profile management should allow for user/password with Jira as well.
+  * Update pyPolarionCli documentation and example adapter for token support.

doc/uml/deployment.puml

@@ -32,17 +32,20 @@ superset -- supersetDb
 
 node "Continuous Integration Server"<<virtual-machine>> {
 
-    package "scripts" {
-        component "pyJiraCli" as pyJiraCli
-        component "pyPolarionCli" as pyPolarionCli
-        component "pySupersetCli" as pySupersetCli
-        component "pyMetricCli" as pyMetricCli
+    package "Scripts" {
+        component "pyJiraCli"
+        component "pyPolarionCli"
+        component "pySupersetCli"
+        component "pyMetricCli"
         component "pyMetricCli_Adapter" as adapter
+        component "pyProfileMgr"
 
         pyJiraCli <.. pyMetricCli: <<uses>>
+        pyJiraCli *--> pyProfileMgr
         pyPolarionCli <.. pyMetricCli: <<uses>>
         pySupersetCli <.. pyMetricCli: <<uses>>
         pyMetricCli *--> adapter
+        pyMetricCli *--> pyProfileMgr
 
         note right of pyMetricCli
             Called by CI cyclic to generate
@@ -55,6 +58,12 @@ node "Continuous Integration Server"<<virtual-machine>> {
             metrics.
         end note
     }
+
+    folder Filesystem {
+        file "Profile data" as profile_data_files
+    }
+
+    pyProfileMgr -- profile_data_files
 }
 
 jiraRestApi )-- pyJiraCli
@@ -75,4 +84,4 @@ note top of polarion
 end note
 
 
-@enduml
+@enduml

examples/adapter/adapter.py

@@ -21,7 +21,7 @@
 #
 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
 # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICU5LAR PURPOSE ARE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
 # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
@@ -62,28 +62,35 @@ class Adapter(AdapterInterface):
         "status_closed": 0
     }
 
-    # Set "filter": "", if you don't want to search in Jira.
+    # Set "filter": "" if you don't want to search in Jira.
     jira_config = {
-        "server": "https://jira.example.com",
+        "profile": "newtec_jira",
+        "server": "",
         "token": "",
-        "filter": "",
+        # Filter to get a specific issue.
+        "filter": "ISSUE=PROJCHANCE-3063",
         "max": "0",  # 0 gets all issues that match the filter.
         "fields": [],
         "full": False
     }
 
-    # Set "query": "", if you don't want to search in Polarion.
+    # Set "query": "" if you don't want to search in Polarion.
     polarion_config = {
+        "profile": "newtec_polarion",
         "username": "",
         "password": "",
-        "server": "http://polarion.example.com/polarion",
-        "project": "",
-        "query": "HAS_VALUE:status",  # Query to get all work items with a status
-        "fields": ["status"]  # Fields to include in the query
+        "server": "",
+        "token": "",
+        "project": "BBRAUN.SPACE2",
+        # Query to get all obsolete work items (shouldn't be many).
+        "query": "severity:Obsolete",
+        # Fields to restrict to.
+        "fields": ["id", "title", "status"],
     }
 
     superset_config = {
-        "server": "http://superset.example.com",
+        "profile": "newtec_superset",
+        "server": "",
         "user": "",
         "password": "",
         "database": 0,  # Primary key of the database

pyproject.toml

@@ -10,7 +10,8 @@ readme = "README.md"
 requires-python = ">=3.9"
 authors = [
     { name = "Gabryel Reyes", email = "gabryel.reyes@newtec.de" },
-    { name = "Juliane Kerpe", email = "juliane.kerpe@newtec.de" }
+    { name = "Juliane Kerpe", email = "juliane.kerpe@newtec.de" },
+    { name = "Stefan Vogel", email = "stefan.vogel@newtec.de" }
 ]
 license = {text = "BSD 3-Clause"}
 classifiers = [
@@ -27,7 +28,8 @@ dependencies = [
     "toml>=0.10.2",
     "pyJiraCli@git+https://github.com/NewTec-GmbH/pyJiraCli",
     "pyPolarionCli@git+https://github.com/NewTec-GmbH/pyPolarionCli",
-    "pySupersetCli@git+https://github.com/NewTec-GmbH/pySupersetCli"
+    "pySupersetCli@git+https://github.com/NewTec-GmbH/pySupersetCli",
+    "pyProfileMgr@git+https://github.com/NewTec-GmbH/pyProfileMgr.git@feature/profile_mgmt_lib"
 ]
 
 [project.optional-dependencies]

setup.py

@@ -19,7 +19,7 @@
 #
 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
 # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICU5LAR PURPOSE ARE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
 # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR

src/pyMetricCli/__main__.py

@@ -20,7 +20,7 @@
 #
 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
 # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICU5LAR PURPOSE ARE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
 # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
@@ -76,7 +76,7 @@
 
 def add_parser() -> argparse.ArgumentParser:
     """ Add parser for command line arguments and
-        set the execute function of each 
+        set the execute function of each
         cmd module as callback for the subparser command.
         Return the parser after all the modules have been registered
         and added their subparsers.
@@ -125,7 +125,7 @@ def _import_adapter(adapter_path: str) -> AdapterInterface:
     adapter_instance = None
 
     if not os.path.isfile(adapter_path):
-        LOG.error("The adapter file does not exist.")
+        LOG.error("The adapter file '%s' does not exist.", adapter_path)
     else:
         module_spec = importlib.util.spec_from_file_location(adapter_name,
                                                              adapter_path)
@@ -134,22 +134,22 @@ def _import_adapter(adapter_path: str) -> AdapterInterface:
         module_spec.loader.exec_module(adapter)
         adapter_instance = adapter.Adapter()
 
-    # Check all required attributes and methods of the adapter class.
-    if not isinstance(adapter_instance, AdapterInterface):
-        LOG.error("The adapter class must inherit from AdapterInterface.")
-        adapter_instance = None
-    else:
-        LOG.info("Adapter class successfully imported.")
+        # Check all required attributes and methods of the adapter class.
+        if not isinstance(adapter_instance, AdapterInterface):
+            LOG.error("The adapter class must inherit from AdapterInterface.")
+            adapter_instance = None
+        else:
+            LOG.info("Adapter class successfully imported.")
 
-        # Check if the values of the output dictionary in the adapter class are unique.
-        output_list = list(adapter_instance.output.keys())
-        output_list_lowercase = [status.lower() for status in output_list]
+            # Check if the values of the output dictionary in the adapter class are unique.
+            output_list = list(adapter_instance.output.keys())
+            output_list_lowercase = [status.lower() for status in output_list]
 
-        number_unique_values = len(set(output_list_lowercase))
-        if number_unique_values != len(output_list):
-            LOG.error(
-                "The keys in the output dictionary in the adapter class must be unique.")
-            adapter_instance = None
+            number_unique_values = len(set(output_list_lowercase))
+            if number_unique_values != len(output_list):
+                LOG.error(
+                    "The keys in the output dictionary in the adapter class must be unique.")
+                adapter_instance = None
 
     return adapter_instance
 
@@ -163,13 +163,13 @@ def _process_jira(adapter: AdapterInterface) -> Ret:
     """
     ret_status = Ret.OK
 
+    # Ignore Jira if filter is empty.
     if adapter.jira_config.get("filter", "") != "":
         # Overwrite the output directory with the temp directory.
         adapter.jira_config["file"] = os.path.join(
             _TEMP_DIR_NAME, "jira_search_results.json")
 
-        LOG.info("Searching in Jira: %s",
-                 adapter.jira_config["filter"])
+        LOG.info("Searching in Jira for '%s'...", adapter.jira_config["filter"])
 
         jira_instance = Jira(adapter.jira_config)
         if jira_instance.is_installed is False:
@@ -192,12 +192,12 @@ def _process_polarion(adapter: AdapterInterface) -> Ret:
     """
     ret_status = Ret.OK
 
+    # Ignore Polarion if query is empty.
     if adapter.polarion_config.get("query", "") != "":
         # Overwrite the output directory with the temp directory.
         adapter.polarion_config["output"] = _TEMP_DIR_NAME
 
-        LOG.info("Searching in Polarion: %s",
-                 adapter.polarion_config["query"])
+        LOG.info("Searching in Polarion: %s", adapter.polarion_config["query"])
 
         polarion_instance = Polarion(adapter.polarion_config)
         if polarion_instance.is_installed is False:

src/pyMetricCli/adapter_interface.py

@@ -20,7 +20,7 @@
 #
 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
 # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICU5LAR PURPOSE ARE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
 # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR

src/pyMetricCli/jira.py

@@ -22,7 +22,7 @@
 #
 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
 # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICU5LAR PURPOSE ARE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
 # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
@@ -98,20 +98,32 @@ def search(self) -> dict:
         Returns:
             dict: Search results.
         """
+
         output = {}
         command_list: list = [
             "search",
-            "--server",
-            self.config["server"],
-            "--token",
-            self.config["token"],
             self.config["filter"],
             "--file",
             self.config["file"],
             "--max",
             self.config["max"],
         ]
 
+        # Append profile arg if profile is set in the 'jira_config'.
+        if "profile" in self.config:
+            command_list += [
+                "--profile",
+                self.config["profile"]
+            ]
+        # Else take the server and token from the 'jira_config'.
+        else:
+            command_list += [
+                "--server",
+                self.config["server"],
+                "--token",
+                self.config["token"],
+            ]
+
         for field in self.config["fields"]:
             command_list.append("--field")
             command_list.append(field)