Merge pull request #307 from Azure/feature-data-plane-support

Support data plane command generation.

docs/pages/faq.md

@@ -13,7 +13,7 @@ weight: 999
 
 ## Does AAZDev support data-plane APIs?
 
-Not Yet. Currently the AAZDev tool is focus on Management-plane APIs. However, the framework of AAZDev is designed to support Data-plane APIs. As we know data-plane APIs are much different cross resource providers. So we plane to support them case by case.
+Yes. We have data-plane supported.
 
 ## Filename too long in Git for Windows
 

docs/pages/usage/workspace_editor.md

@@ -11,7 +11,7 @@ Workspaces are used to save and edit command models before exporting them to `aa
 
 ## Workspace operations
 
-When using aaz-dev from scratch, the workspace editor is the starting point.
+When using aaz-dev from scratch, the workspace editor is the starting point. A workspace should service to only one resource provider in the control/data plane.
 
 ### Create a workspace
 
@@ -35,6 +35,25 @@ Click the `DELETE` button you can delete the opened workspace. It requires to in
 
 ![delete_a_workspace](../../assets/recordings/workspace_editor/delete_a_workspace.gif)
 
+## Data-Plane Client
+
+The data-plane client configuration is required for data-plane commands. It's a resource provider level configuration, which means all data-plane commands in one resource provider will share one configuration.
+
+In the client configuration, the client `endpoint` and `authorization` mechanism should be provided.
+
+In workspace you can set two kinds of `endpoint`:
+
+- Template endpoint: The endpoint follows a special template. The template supports using placeholder which is wrapped by `{}`. For example: `https://{keyVaultName}.vault.azure.net`. Those placeholders will generate arguments, which arg group name is 'Client Args', in all commands.
+- Dynamic endpoint (Coming soon): The endpoint should be retrieved from a property value in a control plane api response.  
+
+For tha `authorization` mechanism, we supports `AAD` auth.
+
+When you try to create command for a new resource-provider of data-plane, the workspace will require you set the client configuration at first.
+![new_client_config](../../assets/recordings/workspace_editor/dataplane_new_client_config.gif)
+
+If you want do some modification in existing client configuration, you can click the `Edit Client Config` button. The change will apply to all the data-plane commands of that resource-provider when you export models from the workspace to aaz.
+![edit_client_config](../../assets/recordings/workspace_editor/dataplane_edit_client_config.gif)
+
 ## Add Swagger Resources
 
 When an empty workspace is opened, the `Add Swagger Resources` page will be prompted out by default.
@@ -124,7 +143,7 @@ The commands in workspace are deleted by the swagger resource, if a resource gen
 
 > **Note**
 >
-> Sometime a command contains multiple resource urls, usually the `list` command. You should delete it multiple times. Because the resources should be removed one by one.
+> Sometimes a command contains multiple resource urls, usually the `list` command. You should delete it multiple times. Because the resources should be removed one by one.
 
 ![delete_the_list_command](../../assets/recordings/workspace_editor/delete_the_list_command.gif)
 
@@ -151,7 +170,7 @@ But sometimes two resources cannot be merged because the `valid part` of the url
 
 ## Modify Help for Commands and Groups
 
-The are two kinds of summaries for commands and command-groups:
+There are two kinds of summaries for commands and command-groups:
 
 - Short Summary: Will be displayed in the help of the parent level and itself. **It's required for all commands and groups**
 - Long Summary: Only be displayed in its help. It's optional. Supports multiple lines.
@@ -309,7 +328,7 @@ When you flatten `--prop-b`, the argument in command level will be:
 
 ### Hidden Arguments
 
-While editing the arguments, you can hidden it. The code of hidden arguments will **NOT** be generated in azure-cli, so the users cannot pass a value for hidden arguments. The command models in aaz will keep the hidden arguments, and you can enable them in the future.  
+While editing the arguments, you can hide it. The code of hidden arguments will **NOT** be generated in azure-cli, so the users cannot pass a value for hidden arguments. The command models in aaz will keep the hidden arguments, and you can enable them in the future.  
 
 ![hidden_arguments](../../assets/recordings/workspace_editor/hidden_arguments.gif)
 

src/aaz_dev/cli/api/_cmds.py

@@ -157,6 +157,7 @@ def generate_by_swagger_tag(profile, swagger_tag, extension_or_module_name, cli_
                 })
 
             v = v_list[0]
+            # TODO: handle plane here
             cfg_reader = aaz_specs.load_resource_cfg_reader(Config.DEFAULT_PLANE, resource_id, v)
             if not cfg_reader:
                 logger.error(f"Command models not exist in aaz for resource: {resource_id} version: {v}")

src/aaz_dev/cli/controller/az_atomic_profile_builder.py

@@ -6,21 +6,24 @@
 
 from cli.model.atomic import CLIAtomicProfile, CLIAtomicCommandGroup, CLIAtomicCommandGroupRegisterInfo, \
     CLIAtomicCommand, CLIAtomicCommandRegisterInfo, CLISpecsResource, CLICommandGroupHelp, CLICommandHelp, \
-    CLICommandExample
+    CLICommandExample, CLIAtomicClient
 from command.controller.cfg_reader import CfgReader
 from command.controller.specs_manager import AAZSpecsManager
 from command.model.configuration import CMDHttpOperation, CMDCommand, CMDArgGroup, CMDObjectOutput, \
     CMDHttpResponseJsonBody, CMDObjectSchemaBase
 from swagger.utils.tools import swagger_resource_path_to_resource_id
 from utils.stage import AAZStageEnum
 from utils.exceptions import ResourceNotFind
+from utils.plane import PlaneEnum
+from utils.case import to_camel_case, to_snake_case
 
 logger = logging.getLogger('backend')
 
 
 class AzAtomicProfileBuilder:
 
-    def __init__(self, by_patch=False):
+    def __init__(self, mod_name, by_patch=False):
+        self._mod_name = mod_name
         self._aaz_spec_manager = AAZSpecsManager()
         self._by_patch = by_patch
 
@@ -29,15 +32,20 @@ def __call__(self, view_profile):
         profile.name = view_profile.name
         if view_profile.command_groups:
             cmd_groups = {}
+            cmd_clients = {}
             for name, view_cmd_group in view_profile.command_groups.items():
-                cmd_group = self._build_command_group(view_cmd_group)
+                cmd_group, clients = self._build_command_group(view_cmd_group)
                 cmd_groups[name] = cmd_group
+                cmd_clients.update(clients)
             profile.command_groups = cmd_groups
+            for client in cmd_clients.values():
+                profile.add_client(client)
         return profile
 
     def _build_command_group(self, view_command_group):
         command_group = self._build_command_group_from_aaz(*view_command_group.names)
         stages = set()
+        cmd_clients = {}
 
         if view_command_group.commands:
             # always load cfg for full generation
@@ -51,10 +59,12 @@ def _build_command_group(self, view_command_group):
                         load_cfg = True
                         break
             for name, view_cmd in view_command_group.commands.items():
-                cmd = self._build_command(view_cmd, load_cfg)
+                cmd, client = self._build_command(view_cmd, load_cfg)
                 if cmd.register_info is not None:
                     stages.add(cmd.register_info.stage)
                 cmds[name] = cmd
+                cmd_clients[(client.plane, client.name)] = client
+
             command_group.commands = cmds
             if load_cfg:
                 command_group.wait_command = self._complete_command_wait_info(command_group)
@@ -67,10 +77,11 @@ def _build_command_group(self, view_command_group):
         if view_command_group.command_groups:
             cmd_groups = {}
             for name, view_cmd_group in view_command_group.command_groups.items():
-                cmd_group = self._build_command_group(view_cmd_group)
+                cmd_group, clients = self._build_command_group(view_cmd_group)
                 if cmd_group.register_info is not None:
                     stages.add(cmd_group.register_info.stage)
                 cmd_groups[name] = cmd_group
+                cmd_clients.update(clients)
             command_group.command_groups = cmd_groups
 
         if AAZStageEnum.Stable in stages:
@@ -84,13 +95,14 @@ def _build_command_group(self, view_command_group):
         else:
             raise NotImplementedError()
 
-        return command_group
+        return command_group, cmd_clients
 
     def _build_command(self, view_command, load_cfg):
         command = self._build_command_from_aaz(*view_command.names, version_name=view_command.version, load_cfg=load_cfg)
         if not view_command.registered:
             command.register_info = None
-        return command
+        client = self._build_client_from_aaz(plane=command.resources[0].plane)
+        return command, client
 
     def _build_command_group_from_aaz(self, *names):
         aaz_cg = self._aaz_spec_manager.find_command_group(*names)
@@ -146,6 +158,24 @@ def _build_command_from_aaz(self, *names, version_name, load_cfg=True):
             command.register_info.confirmation = cmd_cfg.confirmation
         return command
 
+    def _build_client_from_aaz(self, plane):
+        client = CLIAtomicClient({
+            "plane": plane,
+            "name": PlaneEnum.http_client(plane)
+        })
+        if plane in PlaneEnum._config:
+            # use the clients registered in azure/cli/core/aaz/_client.py
+            client.registered_name = client.name
+        else:
+            # generate client based on client config
+            cfg_reader = self._aaz_spec_manager.load_client_cfg_reader(plane)
+            assert cfg_reader, "Missing Client config for '" + plane + "' plane."
+            client.cfg = cfg_reader.cfg
+            scope = PlaneEnum.get_data_plane_scope(plane) or plane
+            client.registered_name = (to_camel_case(f"AAZ {scope.replace('.', ' ')} {client.name}") +
+                                      f'_{to_snake_case(self._mod_name)}')  # for example: AAZAzureCodesigningDataPlaneClient_network
+        return client
+
     @classmethod
     def _complete_command_wait_info(cls, command_group):
         assert command_group.commands

src/aaz_dev/cli/controller/az_client_generator.py

@@ -0,0 +1,37 @@
+from utils import exceptions
+import logging
+from cli.model.atomic import CLIAtomicCommand, CLIAtomicClient
+
+logger = logging.getLogger('backend')
+
+
+class AzClientsGenerator:
+
+    def __init__(self, clients: [CLIAtomicClient]):
+        # only clients with cfg will be generated.
+        self._clients = sorted([client for client in clients if client.cfg], key=lambda c: c.registered_name)
+
+    def iter_clients(self):
+        for client in self._clients:
+            yield AzClientGenerator(client)
+
+    def is_empty(self):
+        return len(self._clients) == 0
+
+
+class AzClientGenerator:
+
+    def __init__(self, client):
+        self._client = client
+
+    @property
+    def registered_name(self):
+        return self._client.registered_name
+
+    def iter_hosts(self):
+        for template in self._client.cfg.endpoints.templates:
+            yield template.cloud, template.template
+
+    @property
+    def aad_scopes(self):
+        return self._client.cfg.auth.aad.scopes

src/aaz_dev/cli/controller/az_command_generator.py

@@ -1,10 +1,9 @@
-from cli.model.atomic import CLIAtomicCommand
+from cli.model.atomic import CLIAtomicCommand, CLIAtomicClient
 from command.model.configuration import CMDCommand, CMDHttpOperation, CMDCondition, CMDConditionAndOperator, \
     CMDConditionOrOperator, CMDConditionNotOperator, CMDConditionHasValueOperator, CMDInstanceUpdateOperation, \
     CMDJsonInstanceUpdateAction, CMDResourceGroupNameArg, CMDJsonSubresourceSelector, CMDInstanceCreateOperation, \
     CMDInstanceDeleteOperation, CMDJsonInstanceCreateAction, CMDJsonInstanceDeleteAction
 from utils.case import to_camel_case, to_snake_case
-from utils.plane import PlaneEnum
 from .az_operation_generator import AzHttpOperationGenerator, AzJsonUpdateOperationGenerator, \
     AzGenericUpdateOperationGenerator, AzRequestClsGenerator, AzResponseClsGenerator, \
     AzInstanceUpdateOperationGenerator, AzLifeCycleInstanceUpdateCallbackGenerator, AzJsonCreateOperationGenerator, \
@@ -123,10 +122,11 @@ class AzCommandGenerator:
 
     ARGS_SCHEMA_NAME = "_args_schema"
 
-    def __init__(self, cmd: CLIAtomicCommand, is_wait=False):
+    def __init__(self, cmd: CLIAtomicCommand, client: CLIAtomicClient, is_wait=False):
         self.cmd = cmd
         self.is_wait = is_wait
         self.cmd_ctx = AzCommandCtx()
+        self.client = client
 
         if cmd.names[-1] in ("create", "list"):
             # disable id part for create and list command
@@ -140,6 +140,10 @@ def __init__(self, cmd: CLIAtomicCommand, is_wait=False):
 
         # prepare arguments
         self.arg_groups = []
+        if self.client.cfg and self.client.cfg.arg_group and self.client.cfg.arg_group.args:
+            # add client args
+            self.arg_groups.append(AzArgGroupGenerator(self.ARGS_SCHEMA_NAME, self.cmd_ctx, self.client.cfg.arg_group))
+
         if self.cmd.cfg.arg_groups:
             for arg_group in self.cmd.cfg.arg_groups:
                 if arg_group.args:
@@ -170,7 +174,8 @@ def __init__(self, cmd: CLIAtomicCommand, is_wait=False):
                 op_cls_name = to_camel_case(operation.operation_id)
                 if operation.long_running:
                     lr = True
-                op = AzHttpOperationGenerator(op_cls_name, self.cmd_ctx, operation)
+                client_endpoints = self.client.cfg.endpoints if self.client.cfg else None
+                op = AzHttpOperationGenerator(op_cls_name, self.cmd_ctx, operation, client_endpoints=client_endpoints)
                 self.http_operations.append(op)
             elif isinstance(operation, CMDInstanceUpdateOperation):
                 if isinstance(operation.instance_update, CMDJsonInstanceUpdateAction):
@@ -289,9 +294,6 @@ def __init__(self, cmd: CLIAtomicCommand, is_wait=False):
             elif resource.plane != self.plane:
                 raise ValueError(f"Find multiple planes in a command: {resource.plane}, {self.plane}")
 
-        self.client_type = PlaneEnum.http_client(self.plane)
-        # TODO: add support for DataPlaneClient client_type
-
         # prepare outputs
         self.outputs = []
         self.paging = False

src/aaz_dev/cli/controller/az_module_manager.py

@@ -72,7 +72,7 @@ def load_module(self, mod_name):
     def update_module(self, mod_name, profiles, **kwargs):
         aaz_folder = self.get_aaz_path(mod_name)
         generators = {}
-        atomic_builder = AzAtomicProfileBuilder(by_patch=kwargs.pop('by_patch', False))
+        atomic_builder = AzAtomicProfileBuilder(mod_name=mod_name, by_patch=kwargs.pop('by_patch', False))
         for profile_name, profile in profiles.items():
             profile = atomic_builder(profile)
             generators[profile_name] = AzProfileGenerator(aaz_folder, profile)

src/aaz_dev/cli/controller/az_operation_generator.py

@@ -41,10 +41,12 @@ def __init__(self, name, variant_key, is_selector_variant):
 
 class AzHttpOperationGenerator(AzOperationGenerator):
 
-    def __init__(self, name, cmd_ctx, operation):
+    def __init__(self, name, cmd_ctx, operation, client_endpoints):
         super().__init__(name, cmd_ctx, operation)
         assert isinstance(self._operation, CMDHttpOperation)
 
+        self.client_endpoints = client_endpoints
+
         if self._operation.long_running is not None:
             self.is_long_running = True
             self.lro_options = {
@@ -128,10 +130,27 @@ def method(self):
 
     @property
     def url_parameters(self):
+        parameters = []
+        # add params in client endpoints
+        if self.client_endpoints and self.client_endpoints.params:
+            for param in self.client_endpoints.params:
+                kwargs = {}
+                if param.skip_url_encoding:
+                    kwargs['skip_quote'] = True
+                if param.required:
+                    kwargs['required'] = param.required
+                arg_key, hide = self._cmd_ctx.get_argument(param.arg)
+                if not hide:
+                    parameters.append([
+                        param.name,
+                        arg_key,
+                        False,
+                        kwargs
+                    ])
+        # add params in client path
         path = self._operation.http.request.path
         if not path:
-            return None
-        parameters = []
+            return parameters or None
         if path.params:
             for param in path.params:
                 kwargs = {}
@@ -161,7 +180,8 @@ def url_parameters(self):
                     True,
                     kwargs
                 ])
-        return parameters
+
+        return parameters or None
 
     @property
     def query_parameters(self):