Merge pull request #106 from oracle/sprint-9

Sprint 9

CHANGELOG.md

@@ -1,10 +1,24 @@
 # Release Notes
 
 
+## Version 0.11.0 
+**Release Date**: 16th September 2020
+### Features
+1. Modify Multiple Select Options to Checkbox Group
+2. Add Option to Query Dialog to allow the user to specify if the query will drill down into Sub-Compartments.
+3. Modify start functionality to allow the user to specify a local directory where the user templates will be saved.
+4. Add display/edit of the Model Title and Description to the web interface.
+
+### Bug Fixes
+1. Database System CPU Count causing error when generating Terraform (Issue: #98)
+2. Network Security Groups, for a VCN, were not deleted when the VCN is deleted.
+3. DRG not being displayed if it was not attached to a VCN following a query.
+
+
 ## Version 0.10.2 
 **Release Date**: 27th August 2020
 ### Bug Fixes
-1. Resolve issue where deleting an artefact would cause an addition OKE cluster to be created if one exists on the canvas. In addition deleteing OKE caused it to be removed from the model but not the view. 
+1. Resolve issue where deleting an artefact would cause an addition OKE cluster to be created if one exists on the canvas. In addition deleting OKE caused it to be removed from the model but not the view. 
 
 
 ## Version 0.10.1 

README.md

@@ -1,26 +1,33 @@
-# OCI Designer Toolkit [0.10.2](CHANGELOG.md#version-0.10.2)
+# OCI Designer Toolkit [0.11.0](CHANGELOG.md#version-0.11.0)
 
-OCI designer toolKIT (OKIT) is a set of tools for enabling design, deploy and visualise OCI environments 
-through a graphical web based interface. 
+OCI designer toolKIT (OKIT) is a browser based tool that allows the user to [design](https://www.ateam-oracle.com/introduction-to-okit-the-oci-designer-toolkit), 
+[deploy](https://www.ateam-oracle.com/introduction-to-okit-the-oci-designer-toolkit) and visualise ([introspect/query](https://www.ateam-oracle.com/the-oci-designer-toolkit-query-feature)) 
+OCI environments through a graphical web based interface. 
 
-The Web based interface will allow architects and designers to build a visual representation of their infrastructure
-and then export this in a number of formats. 
+- [Design](https://www.ateam-oracle.com/introduction-to-okit-the-oci-designer-toolkit)
 
-- svg
-- png
-- jpeg
+    The Web based interface will allow architects and designers to build a visual representation of their infrastructure
+    and then export this in a number of formats. 
 
-Once completed the design can be enhanced to add key property information that will allow the designed infrastructure to
-be exported to a number of DevOps frameworks.
+    - svg
+    - png
+    - jpeg
 
-- Ansible
-- Terraform
-- OCI Resource Manager
+- [Generate](https://www.ateam-oracle.com/introduction-to-okit-the-oci-designer-toolkit)
 
-This allows for rapid proto-typing and building.
+    Once completed the design can be enhanced to add key property information allowing the designed infrastructure to
+    be exported to a number of DevOps frameworks.
+
+    - Ansible
+    - Terraform
+    - OCI Resource Manager
+
+    This allows for rapid proto-typing and building.
 
-OKIT will also all the Operations user to capture existing OCI environments, through simple query functionality embedded in  
-web interface, to provide a portable generic, json, file that can be used to visualise existing systems or generate terraform/ansible. 
+- [Introspect](https://www.ateam-oracle.com/the-oci-designer-toolkit-query-feature)
+
+    OKIT will also allow the user to introspect existing OCI environments, through simple query functionality embedded within the
+    web interface, to provide a portable generic json file, that can be used to visualise existing systems or generate terraform/ansible. 
 
 
 
@@ -30,24 +37,35 @@ Detailed OKIT Installation steps can be found in the [OCI Designer Toolkit Insta
 ### Quick Start
 The docker image is the recommended runtime server OKIT provides a simple Docker Compose script to build and start the container.
 
+#### Clone Repository
+```bash
+git clone --depth 1 https://github.com/oracle/oci-designer-toolkit.git
+```
+
 #### Create Config File
 
-Create the directory __<OKIT_ROOT_DIR>/containers/oci__ and within it a file called __config__ with contents similar to
-that below.
+Create the OCI cli __config__ file in the directory __oci-designer-toolkit/containers/oci__ with contents similar to that below.
+The __*key_file*__ is a fixed value because the contents of the __oci-designer-toolkit/containers/oci__ will be copied to the
+appropriate users home directory, as __~/.oci__, during the build process.
 
 ```properties
 [DEFAULT]
 user=ocid1.user.oc1..aaaaaaaak6z......
 fingerprint=3b:7e:37:ec:a0:86:1....
-key_file=~/.oci/oci_api_key.pem
+key_file=~/.oci/oci_api_key.pem  
 tenancy=ocid1.tenancy.oc1..aaaaaaaawpqblfem........
 region=us-phoenix-1
 ```
+
+#### Copy Key File
+
+Copy your __oci_api_key.pem__ to the __oci-designer-toolkit/containers/oci__ directory.
+
 #### Build and Start
 
 ```bash
 cd oci-designer-toolkit/containers/docker
-docker-compose up
+docker-compose up --detach
 ```
 
 __*Note:*__ For full connection to OCI you will need to follow the [Installation Create Config File instruction](documentation/Installation.md#oci-config-file) 

containers/docker/Dockerfile

@@ -5,7 +5,7 @@
 FROM oraclelinux:7-slim
 LABEL "provider"="Oracle" \
       "issues"="https://github.com/oracle/oci-designer-toolkit/issues" \
-      "version"="0.9.2" \
+      "version"="0.11.0" \
       "description"="OKIT Web Server Container." \
       "copyright"="Copyright (c) 2020, Oracle and/or its affiliates."
 SHELL ["/bin/bash", "-c"]
@@ -43,7 +43,9 @@ RUN yum install -y \
         oci==2.6.0 \
         pyyaml==5.2 \
 # Create Workspace
- && mkdir -p /okit/{config,log,visualiser,okitweb,workspace}
+ && mkdir -p /okit/{config,log,visualiser,okitweb,workspace,templates} \
+ && mkdir -p /okit/okitweb/static/okit/templates \
+ && ln -s /okit/templates /okit/okitweb/static/okit/templates/user
 # Copy source code
 COPY okitweb /okit/okitweb
 COPY visualiser /okit/visualiser

containers/docker/docker-compose.yml

@@ -13,4 +13,6 @@ services:
     ports:
       - 443:443/tcp
       - 80:80/tcp
+    volumes:
+      - "~/okit/user/templates:/okit/templates"
 ...

containers/vagrant/Vagrantfile

@@ -33,6 +33,8 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
     # Disable Audio to prevent CPU spikes
     v.customize ["modifyvm", :id, "--audio", "none"]
   end
+  # This will expose the local OKIT Template folder within the VM
+  config.vm.synced_folder "~/okit/user/templates", "/okit/templates"
 
   # VM hostname
   config.vm.hostname = NAME

documentation/Installation.md

@@ -20,40 +20,73 @@ python modules are installed and in addition provide a simple flask server that
 
 
 ## Clone Repository
-Before the building either the Docker or Vagrant Images the project will nee to be cloned from the Git Repository (or downloaded)
-and it is recommended that the latest Stable Release be cloned. The latest stable version number if shown in the README
-and the associated Release tag is in the format vX.Y.Z hence for the version 0.10.2 the Release tag will be 
-**v0.10.2**. The command shows how this can be cloned to the local machine.
+Before the building either the Docker or Vagrant Images the project will need to be cloned, or downloaded, from the GitHub 
+Repository (or downloaded). The master branch is always the latest Stable Release but previous releases can be found using 
+associated Release tag is in the format vX.Y.Z hence for the version 0.11.0 the Release tag will be **v0.11.0**. 
+
+The command shows how this can be cloned to the local machine.
 
 ```bash
-git clone -b v0.10.2 --depth 1 git@github.com:oracle/oci-designer-toolkit.git
+git clone --depth 1 git@github.com:oracle/oci-designer-toolkit.git
 ```
 
 or 
 
 ```bash
-git clone -b v0.10.2 --depth 1 https://github.com/oracle/oci-designer-toolkit.git
+git clone --depth 1 https://github.com/oracle/oci-designer-toolkit.git
 ```
 
 ### Download
 If you do not have git installed locally the current release of OKIT can be retrieved by downloading it as a zip file from
-https://github.com/oracle/oci-designer-toolkit/archive/v0.10.2.zip
+[https://github.com/oracle/oci-designer-toolkit/archive/master.zip](https://github.com/oracle/oci-designer-toolkit/archive/master.zip)
+
+### Update
+If you have previously cloned the GitHub Repository you can update to the latest release by pulling it from the repository 
+with the following command.
+
+```bash
+cd oci-designer-toolkit
+git pull
+```
+
+Your Docker / Vangrant image will need to be rebuilt using one of the following commands:
+
+#### Docker Compose
+```bash
+cd oci-designer-toolkit/containers/docker
+docker-compose stop
+docker-compose build
+```
 
+#### Docker
+```bash
+cd oci-designer-toolkit
+docker rmi okit
+docker build --tag okit --file ./containers/docker/Dockerfile --force-rm .
+```
+
+#### Vagrant
+```bash
+cd oci-designer-toolkit/containers/vagrant
+vagrant halt
+vagrant destroy -f
+vagrant up
+vangrant reload
+```
 
 ## OCI Config File
 Before executing any of the docker container scripts OKIT requires an OCI connection configuration file. This file 
 contains the connection information used by OKIT when executing queries or exporting to Resource Manager.
 
-__*Note:*__ The key_file entry __must not__ be an Absolute path on the host machine. The config directory will be mounted
-to the docker, linux, root user ~/.oci directory.
+__*Note:* The key_file entry *must not* be an Absolute path on the host machine. The config directory will be copied
+to the docker linux root user ~/.oci directory which will be created during the build.__
 
 If you have already installed the OCI SDK/CLI on you machine then you will have already created this file. If you do not 
 have the sdk or cli installed then we will need to create a config as defined in the next section.
 
 ### Creating the Config File
 
-Create the directory __<OKIT_ROOT_DIR>/containers/oci__ and within it a file called __config__ with contents similar to
-that below.
+Create the OCI cli __config__ file in the directory __<OKIT_ROOT_DIR>/containers/oci__ with contents similar to that below.
 
 ```properties
 [DEFAULT]
@@ -74,7 +107,7 @@ The docker image is the recommended runtime server OKIT provides a simple Docker
 
 ```bash
 cd oci-designer-toolkit/containers/docker
-docker-compose up
+docker-compose up --detach
 ```
 
 ## Docker 
@@ -103,6 +136,7 @@ cd oci-designer-toolkit
 docker run -d --rm -p 443:443 -p 80:80 \
            --name okit \
            --hostname okit \
+           --volume ~/okit/user/templates:/okit/templates \
            okit
 ```
 
@@ -122,7 +156,7 @@ If you want to run the image in and interactive mode then replace to _-d_ in the
 2. Install [Vagrant](https://vagrantup.com/)
 
 ### Copy the OCI_CONFIG_DIR folder 
-Copy the contents of the OCI_CONFIG_DIR directory to the oci-designer-toolkit/containers/vagrant/__oci__ folder. 
+Copy the contents of the OCI_CONFIG_DIR directory to the oci-designer-toolkit/containers/__oci__ folder. 
 
 The vagrant should now have these folders & files: 
 - Vagrantfile
@@ -131,7 +165,7 @@ The vagrant should now have these folders & files:
 ### Build Vagrant Image
 ```bash
 cd oci-designer-toolkit/containers/vagrant/
-vagrant up; vagrant ssh
+vagrant up; vagrant reload; vagrant ssh
 ```
 **NOTE**: This step takes about 30 minutes on my mac when you build the VM, a little longer the first time as the Vbox image 
 is downloaded from github. Once the VM is built the vagrant up should just take a few seconds.

documentation/Usage.md

@@ -56,6 +56,7 @@ cd oci-designer-toolkit
 docker run -d --rm -p 443:443 -p 80:80 \
            --name okit \
            --hostname okit \
+           --volume ~/okit/user/templates:/okit/templates \
            okit
 ```
 The __<OKIT_ROOT_DIR>/containers/scripts__ contains helper scripts for Linux/Mac and Windows PowerShell.

okitweb/okitImport.py

@@ -11,9 +11,9 @@
 # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
 
 import os
+import urllib
 from flask import Blueprint
 from flask import request
-
 import json
 from common.okitCommon import logJson
 from common.okitLogging import getLogger
@@ -22,17 +22,22 @@
 # Configure logging
 logger = getLogger()
 
-bp = Blueprint('parsers', __name__, url_prefix='/okit/import', static_folder='static/okit')
+bp = Blueprint('parsers', __name__, url_prefix='/okit/parse', static_folder='static/okit')
 
 debug_mode = bool(str(os.getenv('DEBUG_MODE', 'False')).title())
 
-@bp.route('hcljson', methods=(['POST']))
-def importHclJson():
-    logger.debug('JSON : {0:s}'.format(str(request.json)))
-    if request.method == 'POST':
-        logJson(request.json)
+@bp.route('hcljson', methods=(['GET']))
+def parseHclJson():
+    #logger.debug('JSON : {0:s}'.format(str(request.json)))
+    if request.method == 'GET':
+        query_string = request.query_string
+        parsed_query_string = urllib.parse.unquote(query_string.decode())
+        query_json = json.loads(parsed_query_string)
+        logJson(query_json)
         # Import HCL
-        parser = OkitHclJsonParser(request.json)
-        return json.dumps(parser.parse(), sort_keys=False, indent=2, separators=(',', ': '))
+        parser = OkitHclJsonParser(query_json)
+        response_json = parser.parse()
+        logJson(response_json)
+        return json.dumps(response_json, sort_keys=False, indent=2, separators=(',', ': '))
     else:
         return '404'

okitweb/okitOci.py

@@ -38,6 +38,7 @@
 from facades.ociFileStorageSystems import OCIFileStorageSystems
 from facades.ociImage import OCIImages
 from facades.ociInstance import OCIInstances
+from facades.ociInstancePool import OCIInstancePools
 from facades.ociInternetGateway import OCIInternetGateways
 from facades.ociLoadBalancer import OCILoadBalancers
 from facades.ociLocalPeeringGateway import OCILocalPeeringGateways
@@ -229,6 +230,10 @@ def ociArtifacts(artifact):
         logger.info('---- Processing Instances')
         oci_instances = OCIInstances(config=config, profile=config_profile, compartment_id=query_json['compartment_id'])
         response_json = oci_instances.list(filter=query_json.get('instance_filter', None))
+    elif artifact == 'InstancePool':
+        logger.info('---- Processing Instance Pools')
+        oci_instance_pools = OCIInstancePools(config=config, profile=config_profile, compartment_id=query_json['compartment_id'])
+        response_json = oci_instance_pools.list(filter=query_json.get('instance_filter', None))
     elif artifact == 'InternetGateway':
         logger.info('---- Processing Internet Gateways')
         oci_internet_gateways = OCIInternetGateways(config=config, profile=config_profile, compartment_id=query_json['compartment_id'], vcn_id=query_json['vcn_id'])

okitweb/okitWebDesigner.py

@@ -159,7 +159,7 @@ def designer():
     template_dirs = {}
     logger.debug('Walking the template directories')
     rootdir = os.path.join(bp.static_folder, 'templates')
-    for (dirpath, dirnames, filenames) in os.walk(rootdir):
+    for (dirpath, dirnames, filenames) in os.walk(rootdir, followlinks=True):
         logger.debug('dirpath : {0!s:s}'.format(dirpath))
         logger.debug('dirnames : {0!s:s}'.format(dirnames))
         logger.debug('filenames : {0!s:s}'.format(filenames))

okitweb/static/okit/css/okit_designer.css

@@ -184,6 +184,30 @@
     overflow: scroll;
 }
 
+.okit-canvas-details {
+    display: block;
+    padding-bottom: 5px;
+    width: 100%;
+    overflow: hidden;
+}
+
+.okit-canvas-details > div > div > label {
+    font-family: Helvetica Neue,Helvetica,Arial,sans-serif;
+    font-weight: bold;
+    font-size: small;
+    margin-left: 1em;
+}
+
+.okit-canvas-details > div > div > input {
+    font-family: Helvetica Neue,Helvetica,Arial,sans-serif;
+    font-weight: normal;
+    margin-left: 1em;
+}
+
+.okit-canvas-details > div {
+    overflow: hidden;
+}
+
 /*
 ** Property Sheet (Right Column) Classes
  */

okitweb/static/okit/fragments/json/bastion_server.json

@@ -44,7 +44,7 @@
       ],
       "source_details": {
         "os": "Oracle Linux",
-        "version": "7.7",
+        "version": "",
         "boot_volume_size_in_gbs": "50",
         "source_type": "image"
       },