Merge pull request #454 from oracle/v0.26.0-sprint-26

V0.26.0 sprint 26
oracle · Sep 8, 2021 · 0a593c6 · 0a593c6
2 parents af039e7 + 722e01b
commit 0a593c6
Show file tree

Hide file tree

Showing 129 changed files with 2,037 additions and 374 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,18 @@
 # Release Notes
 
 
+## Version 0.26.0
+**Release Date**: 8th September 2021
+### Features
+1. Temporarily deprecate Ansible export by making it command line flag driven (ansible=true)
+2. Add Policy Resource to Palette and allow the user to drag them onto the canvas. It assumes that the user understands the policy statement syntax. All policies will be created within the users home region.
+
+### Bug Fixes
+1. GitHub Issue #440 : Cannot generate terraform when NSG have Rules.
+2. Fix issue where DB System were always created in the root deployment compartment not specified sub-compartment.
+3. GitHub Issue #447 : Unable to create Object Storage Bucket with OKIT
+
+
 ## Version 0.25.1
 **Release Date**: 20th August 2021
 ### Bug Fixes

diff --git a/Dockerfile b/Dockerfile
@@ -5,7 +5,7 @@
 FROM oraclelinux:7-slim
 LABEL "provider"="Oracle" \
       "issues"="https://github.com/oracle/oci-designer-toolkit/issues" \
-      "version"="0.25.1" \
+      "version"="0.26.0" \
       "description"="OKIT Web Server Container." \
       "copyright"="Copyright (c) 2020, 2021, Oracle and/or its affiliates."
 SHELL ["/bin/bash", "-c"]

diff --git a/README.md b/README.md
@@ -1,4 +1,4 @@
-# Oracle Cloud Infrastructure Designer and Visualisation Toolkit [0.25.1](CHANGELOG.md#version-0.25.1)
+# Oracle Cloud Infrastructure Designer and Visualisation Toolkit [0.26.0](CHANGELOG.md#version-0.26.0)
 
 ```diff
 + The 0.25.0 release has now moved the templates from the menu to a new side panel and the can be opened by double clicking 

diff --git a/containers/docker/Dockerfile b/containers/docker/Dockerfile
@@ -5,7 +5,7 @@
 FROM oraclelinux:7-slim
 LABEL "provider"="Oracle" \
       "issues"="https://github.com/oracle/oci-designer-toolkit/issues" \
-      "version"="0.25.1" \
+      "version"="0.26.0" \
       "description"="OKIT Web Server Container." \
       "copyright"="Copyright (c) 2020, 2021, Oracle and/or its affiliates."
 SHELL ["/bin/bash", "-c"]
@@ -41,7 +41,7 @@ RUN yum install -y \
 # Upgrade pip
  && python3 -m pip install --upgrade pip \
 # Install required python modules
- && pip3 install --no-cache-dir \
+ && python3 -m pip install --no-cache-dir \
         authlib==0.15.3 \
         flask==1.1.2 \
         gitpython==3.1.14 \

diff --git a/containers/services/flask.service b/containers/services/flask.service
@@ -10,6 +10,7 @@ Type=simple
 # the specific user that our service will run as
 User=root
 Group=root
+Environment="OKIT_DIR=/okit"
 Environment="PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"
 Environment="PYTHONPATH=:/okit/visualiser:/okit/okitweb:/okit"
 Environment="FLASK_APP=okitweb"

diff --git a/containers/services/gunicorn.http.service b/containers/services/gunicorn.http.service
@@ -0,0 +1,27 @@
+# Copyright (c) 2020, Oracle and/or its affiliates.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
+
+[Unit]
+Description=gunicorn daemon
+#Requires=gunicorn.socket
+After=network.target
+
+[Service]
+Type=simple
+# the specific user that our service will run as
+User=root
+Group=root
+Environment="OKIT_DIR=/okit"
+Environment="PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"
+Environment="PYTHONPATH=:/okit/visualiser:/okit/okitweb:/okit"
+RuntimeDirectory=gunicorn
+WorkingDirectory=/okit/workspace
+ExecStart=/usr/local/bin/gunicorn okitweb.wsgi:app --config /okit/config/gunicorn_http.py
+ExecReload=/bin/kill -s HUP $MAINPID
+KillMode=mixed
+TimeoutStopSec=5
+TimeoutStartSec=300
+PrivateTmp=true
+
+[Install]
+WantedBy=multi-user.target
diff --git a/containers/services/gunicorn.https.service b/containers/services/gunicorn.https.service
@@ -0,0 +1,27 @@
+# Copyright (c) 2020, Oracle and/or its affiliates.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
+
+[Unit]
+Description=gunicorn daemon
+#Requires=gunicorn.socket
+After=network.target
+
+[Service]
+Type=simple
+# the specific user that our service will run as
+User=root
+Group=root
+Environment="OKIT_DIR=/okit"
+Environment="PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"
+Environment="PYTHONPATH=:/okit/visualiser:/okit/okitweb:/okit"
+RuntimeDirectory=gunicorn
+WorkingDirectory=/okit/workspace
+ExecStart=/usr/local/bin/gunicorn okitweb.wsgi:app --config /okit/config/gunicorn_https.py
+ExecReload=/bin/kill -s HUP $MAINPID
+KillMode=mixed
+TimeoutStopSec=5
+TimeoutStartSec=300
+PrivateTmp=true
+
+[Install]
+WantedBy=multi-user.target
diff --git a/containers/services/gunicorn.service → containers/services/gunicorn.oci.service b/containers/services/gunicorn.service → containers/services/gunicorn.oci.service
@@ -11,6 +11,7 @@ Type=simple
 # the specific user that our service will run as
 User=root
 Group=root
+Environment="OKIT_DIR=/okit"
 Environment="PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"
 Environment="PYTHONPATH=:/okit/visualiser:/okit/okitweb:/okit"
 Environment="OCI_CLI_AUTH=instance_principal"

diff --git a/documentation/Development.md b/documentation/Development.md
@@ -1,12 +1,12 @@
-# OCI Designer Toolkit Artefact (Resource) Development
+# OCI Designer Toolkit Resource (Artefact) Development
 The following guide will take you through a step by step process for developing a new Palette Resource / Artefact for use 
 within OKIT. To implement the new Resource / Artefact a number of new Artefact specific files will need to created whilst 
 integration will require modification of core Javascript files.
 
 ## Table of Contents
 
 1. [Naming Convention](#naming-convention)
-2. [Artefact (Resource) Files](#artefact-resource-files)
+2. [Resource (Artefact) Files](#artefact-resource-files)
 3. [New Files](#new-files)
     1. [Frontend Files](#frontend-files)
     2. [Backend Files](#backend-files)
@@ -24,12 +24,12 @@ All files must be named as per artifact name with the spaces replaced by undersc
 to this is the palette SVG where title case should be used instead of lower case. The reason for this is that the palette
 file name will be manipulated (removing the underscore) and used to dynamically reference all Javascript function names.  
 
-## Artefact (Resource) Files
+## Resource (Artefact) Files
 To add an Artefact to OKIT you will need to create number of files that provide the core functionality associated with the 
 Artefact. In addition to fully integrated the new Artefact into the BUI a number of the core JavaScript / Python scripts will
 need to be updated. The remainder of this document will describe the functionality / files that must be implemented. 
 
-For the worked example we will create the ***Block Storage Volume*** Artefact (Resource) and thus describe how the files
+For the worked example we will create the ***Block Storage Volume*** Resource (Artefact) and thus describe how the files
 are generated and then subsequently modified.
 
 ## New Files
@@ -56,8 +56,8 @@ python3 okitCodeSkeletonGenerator.py --name "<New Artefact Name>"
 ```bash
 python3 okitCodeSkeletonGenerator.py --name "Block Storage Volume"
 INFO: Writing File: ../okitweb/static/okit/model/js/artefacts/block_storage_volume.js
-INFO: Writing File: ../okitweb/static/okit/view/js/artefacts/block_storage_volume.js
-INFO: Writing File: ../okitweb/static/okit/palette/hidden/Block_Storage_Volume.svg
+INFO: Writing File: ../okitweb/static/okit/view/designer/js/artefacts/block_storage_volume.js
+INFO: Writing File: ../okitweb/static/okit/palette/svg/Block_Storage_Volume.svg
 INFO: Writing File: ../okitweb/templates/okit/propertysheets/block_storage_volume.html
 INFO: Writing File: ../okitweb/templates/okit/valueproposition/block_storage_volume.html
 INFO: Writing File: ../visualiser/facade/ociBlockStorageVolume.py
@@ -256,7 +256,7 @@ OkitJson.prototype.deleteBlockStorageVolume = function(id) {
 
 ### Artefact View JavaScript
 #### Generated Skeleton
-Following generation a view file will be created in the **okitweb/static/okit/view/js/artefacts** directory as shown below.
+Following generation a view file will be created in the **okitweb/static/okit/view/designer/js/artefacts** directory as shown below.
 ```javascript
 /*
 ** Copyright (c) 2020, 2021, Oracle and/or its affiliates.
@@ -475,7 +475,7 @@ OkitJsonView.prototype.cloneBlockStorageVolume = function(obj) {
 
 ### Palette SVG
 #### Generated Skeleton
-Following generation an empty SVG file will be created in the **okitweb/static/okit/palette/hidden** directory as shown below. 
+Following generation an empty SVG file will be created in the **okitweb/static/okit/palette/svg** directory as shown below. 
 Because this is created in the hidden directory it will not be displayed in the Palette by default but once modified it 
 should be moved to the appropriate palette subdirectory.
 ```svg
@@ -781,171 +781,22 @@ this can be found in the various, existing, ansible templates.
 ```
 
 ## Updated Files
-Once the core files for the new Artefact (Resource) have been created the developer will need to integrate these into the
-existing functionality by editting a number of common script files. The required edits will be described in the following sections. 
+Once the core files for the new Resource (Artefact) have been created the developer will need to integrate these into the
+existing functionality by editing a number of common script files. The required edits will be described in the following sections. 
 ### Frontend Files
-- [OKIT View JavaScript](#okit-view-javascript)
-- [Designer View JavaScript](#designer-view-javascript)
-- [OKIT OCI Flask Python](#okit-oci-flask-python)
-- [OCI Query JavaScript](#oci-query-javascript)
 
 ### Backend Files
-- [Connection Python](#connection-python)
+- [OCI Query Python](#oci-query-python)
 - [Generator Python](#generator-python)
 
-### OKIT View JavaScript
-The ** okitweb/static/okit/view/js/okit_view.js** will need to be modified to include the new Artefact (Resource) in the clear() and 
-load() function. For our worked example the modifications will be as follows.
-```javascript
-    clear() {
-        this.compartments = [];
-        this.autonomous_databases = [];
-        this.block_storage_volumes = [];
-        this.customer_premise_equipments = [];
-        this.database_systems = [];
-        this.dynamic_routing_gateways = [];
-        this.fast_connects = [];
-        this.file_storage_systems = [];
-        this.instances = [];
-        this.instance_pools = [];
-        this.internet_gateways = [];
-        this.ipsec_connections = [];
-        this.load_balancers = [];
-        this.local_peering_gateways = [];
-        this.mysql_database_systems = [];
-        this.nat_gateways = [];
-        this.network_security_groups = [];
-        this.object_storage_buckets = [];
-        this.oke_clusters = [];
-        this.remote_peering_connections = [];
-        this.route_tables = [];
-        this.security_lists = [];
-        this.service_gateways = [];
-        this.subnets = [];
-        this.virtual_cloud_networks = [];
-        this.virtual_network_interfaces = [];
-    }
-
-    load() {
-        this.clear();
-        for (let artefact of this.okitjson.compartments) {this.newCompartment(artefact);}
-        for (let artefact of this.okitjson.autonomous_databases) {this.newAutonomousDatabase(artefact);}
-        for (let artefact of this.okitjson.block_storage_volumes) {this.newBlockStorageVolume(artefact);}
-        for (let artefact of this.okitjson.customer_premise_equipments) {this.newCustomerPremiseEquipment(artefact);}
-        for (let artefact of this.okitjson.database_systems) {this.newDatabaseSystem(artefact);}
-        for (let artefact of this.okitjson.dynamic_routing_gateways) {this.newDynamicRoutingGateway(artefact);}
-        for (let artefact of this.okitjson.fast_connects) {this.newFastConnect(artefact);}
-        for (let artefact of this.okitjson.file_storage_systems) {this.newFileStorageSystem(artefact);}
-        for (let artefact of this.okitjson.instances) {this.newInstance(artefact);}
-        for (let artefact of this.okitjson.instance_pools) {this.newInstancePool(artefact);}
-        for (let artefact of this.okitjson.internet_gateways) {this.newInternetGateway(artefact);}
-        for (let artefact of this.okitjson.ipsec_connections) {this.newIPSecConnection(artefact);}
-        for (let artefact of this.okitjson.load_balancers) {this.newLoadBalancer(artefact);}
-        for (let artefact of this.okitjson.local_peering_gateways) {this.newLocalPeeringGateway(artefact);}
-        for (let artefact of this.okitjson.mysql_database_systems) {this.newMySQLDatabaseSystem(artefact);}
-        for (let artefact of this.okitjson.nat_gateways) {this.newNATGateway(artefact);}
-        for (let artefact of this.okitjson.network_security_groups) {this.newNetworkSecurityGroup(artefact);}
-        for (let artefact of this.okitjson.object_storage_buckets) {this.newObjectStorageBucket(artefact);}
-        for (let artefact of this.okitjson.oke_clusters) {this.newOkeCluster(artefact);}
-        for (let artefact of this.okitjson.remote_peering_connections) {this.newRemotePeeringConnection(artefact);}
-        for (let artefact of this.okitjson.route_tables) {this.newRouteTable(artefact);}
-        for (let artefact of this.okitjson.security_lists) {this.newSecurityList(artefact);}
-        for (let artefact of this.okitjson.service_gateways) {this.newServiceGateway(artefact);}
-        for (let artefact of this.okitjson.subnets) {this.newSubnet(artefact);}
-        for (let artefact of this.okitjson.virtual_cloud_networks) {this.newVirtualCloudNetwork(artefact);}
-    }
-```
-### Designer View JavaScript
-The ** okitweb/static/okit/view/designer/js/okit_view.js** will need to be modified to include the new Artefact (Resource) in the
-draw() function. This code must be placed within the function at an appropriate position after its parent and before its children.
-For our example we would add the following block after the Compartments have been created.
-```javascript
-        // Block Storage Volumes
-        for (let block_storage_volume of this.block_storage_volumes) {
-            block_storage_volume.draw();
-        }
-```
-### OKIT OCI Flask Python
-The Flask endpoint Python file **okitweb/okitOci.py** will need to be modified to allow for the querying of the new Artefact (Resource).
-This will require that a new condition be implemented within the **ociArtifacts(artifact)** endpoint function that calls the
-list method defined within the [Artefact Python Facade](#artefact-python-facade). For our worked example this will be as follows.
-```python
-    elif artifact == 'BlockStorageVolume':
-        logger.info('---- Processing Block Storage Volumes')
-        oci_block_storage_volumes = OCIBlockStorageVolumes(config=config, profile=config_profile, compartment_id=query_json['compartment_id'])
-        response_json = oci_block_storage_volumes.list(filter=query_json.get('block_storage_volume_filter', None))
-```
-### OCI Query JavaScript
-To facilitate querying of the new Artefact (Resource) we will need to modify the **okitweb/static/okit/query/oci/js/okit_query.js** file 
-to include a new query method and update the parents SubComponent query code. For our worked exmple this will mean modifying the
-queryCompartmentSubComponents(request).
-#### New Query Method 
-```javascript
-    queryBlockStorageVolumes(request) {
-        console.info('------------- Block Storage Volume Query --------------------');
-        console.info('------------- Compartment : ' + request.compartment_id);
-        let me = this;
-        this.region_query_count[request.region]++;
-        $.ajax({
-            type: 'get',
-            url: 'oci/artefacts/BlockStorageVolume',
-            dataType: 'text',
-            contentType: 'application/json',
-            data: JSON.stringify(request),
-            success: function(resp) {
-                let response_json = JSON.parse(resp);
-                regionOkitJson[request.region].load({block_storage_volumes: response_json});
-                if (request.refresh) {okitJsonView.draw();}
-            },
-            error: function(xhr, status, error) {
-                console.warn('Status : ' + status);
-                console.warn('Error  : ' + error);
-            },
-            complete: function () {
-                me.region_query_count[request.region]-- && me.isComplete();
-            }
-        });
-    }
-```
-#### Update Parent SubComponents Method
-```javascript
-    queryCompartmentSubComponents(request) {
-        if (request.sub_compartments) {
-            this.queryCompartments(request);
-        }
-        this.queryVirtualCloudNetworks(request);
-        this.queryBlockStorageVolumes(request);
-        this.queryCustomerPremiseEquipments(request);
-        this.queryDynamicRoutingGateways(request);
-        this.queryAutonomousDatabases(request);
-        this.queryObjectStorageBuckets(request);
-        this.queryFastConnects(request);
-        this.queryInstances(request);
-        this.queryInstancePools(request);
-        this.queryIPSecConnections(request);
-        this.queryRemotePeeringConnections(request);
-        this.queryDatabaseSystems(request);
-        this.queryMySQLDatabaseSystems(request);
-        this.queryFileStorageSystems(request);
-        this.queryOkeClusters(request);
-    }
-```
-### Connection Python
-The visualiser/facades/ociConnection python file will need to be modified to include the new Connection class used within 
-[Artefact Python Facade](#artefact-python-facade). This will need to define a connect method that uses the appropriate API
-client class.
-```python
-class OCIBlockStorageVolumeConnection(OCIConnection):
-    def __init__(self, config=None, configfile=None, profile=None):
-        super(OCIBlockStorageVolumeConnection, self).__init__(config=config, configfile=configfile, profile=profile)
+### OCI Query Python
+To facilitate querying of the new Resource (Artefact) we will need to modify the **visualiser/query/ociQuery.py** file 
+to include the new Resource in the **SUPPORTED_RESOURCES** list and the **DISCOVERY_OKIT_MAP** object. The correct name 
+to be entered into the **SUPPORTED_RESOURCES** can be identified by looking at the **visualiser/discovery/oci_discovery_client.py** 
+whilst the entry in the **DISCOVERY_OKIT_MAP** will be used to map the name to the OKIT model list.
 
-    def connect(self):
-        self.client = oci.core.BlockstorageClient(config=self.config, signer=self.signer)
-        return
-
-```
 ### Generator Python
-The visualiser/generators/ociGenerator python code will need to be edited to include a Render method for the new Artefact (Resource)
+The **visualiser/generators/ociGenerator.py** python code will need to be edited to include a Render method for the new Resource (Artefact)
 and the existing **generate()** method will need to be modified to call the new Render method.
 
 Although the sequence in which resource creation occurs does not matter for terraform it does for other language such as ansible, python or bash.