Merge pull request #208 from oracle/sprint-14

Sprint 14

CHANGELOG.md

@@ -1,5 +1,18 @@
 # Release Notes
 
+## Version 0.16.0
+**Release Date**: 20th January 2021
+### Features
+1. Description now displayed as a right side panel allowing multi line entry (Enhancement Request #165)
+2. Remove dialog associated with Save As Template because the Title & Description are now part of the main screen.
+3. Add Preference to display the OCID for a Resource.
+4. Simple GIT integration for saving Template / Terraform / Ansible files.
+5. Export to Markdown to provide design documentation, description field (Enhancement Request #165) allows Markdown whilst each contains a Documentation field to provide resource specific information.
+
+### Bug Fixes
+1. Object Storage Bucket not displaying queried name (Issue #206)
+
+
 ## Version 0.15.0 
 **Release Date**: 9th December 2020
 ### Features

Dockerfile

@@ -5,7 +5,7 @@
 FROM oraclelinux:7-slim
 LABEL "provider"="Oracle" \
       "issues"="https://github.com/oracle/oci-designer-toolkit/issues" \
-      "version"="0.15.0" \
+      "version"="0.16.0" \
       "description"="OKIT Web Server Container." \
       "copyright"="Copyright (c) 2020, Oracle and/or its affiliates."
 SHELL ["/bin/bash", "-c"]
@@ -43,8 +43,10 @@ RUN yum install -y \
  && pip3 install --no-cache-dir \
         flask==1.1.1 \
         gitpython==3.1.11 \
+        git-url-parse==1.2.2 \
         gunicorn==20.0.4 \
         oci==2.22.0 \
+        openpyxl==3.0.5 \
         pandas==1.1.2 \
         python-magic==0.4.18 \
         pyyaml==5.3.1 \

README.md

@@ -1,4 +1,4 @@
-# Oracle Cloud Infrastructure Designer and Visualisation Toolkit [0.15.0](CHANGELOG.md#version-0.15.0)
+# Oracle Cloud Infrastructure Designer and Visualisation Toolkit [0.16.0](CHANGELOG.md#version-0.16.0)
 
 OCI designer and visualisation toolKIT (OKIT) is a browser based tool that allows the user to [design](https://www.ateam-oracle.com/introduction-to-okit-the-oci-designer-toolkit), 
 [deploy](https://www.ateam-oracle.com/introduction-to-okit-the-oci-designer-toolkit) and visualise ([introspect/query](https://www.ateam-oracle.com/the-oci-designer-toolkit-query-feature)) 
@@ -13,22 +13,30 @@ OCI environments through a graphical web based interface.
     - png
     - jpeg
 
-- [Generate](https://www.ateam-oracle.com/introduction-to-okit-the-oci-designer-toolkit)
+- [Export](https://www.ateam-oracle.com/introduction-to-okit-the-oci-designer-toolkit)
 
     Once completed the design can be enhanced to add key property information allowing the designed infrastructure to
-    be exported to a number of DevOps frameworks.
+    be exported to a number of DevOps frameworks or Markdown for documentation.
 
     - Ansible
     - Terraform
     - OCI Resource Manager
+    - Markdown
 
     This allows for rapid proto-typing and building.
 
 - [Introspect](https://www.ateam-oracle.com/the-oci-designer-toolkit-query-feature)
 
     OKIT will also allow the user to introspect existing OCI environments, through simple query functionality embedded within the
-    web interface, to provide a portable generic json file, that can be used to visualise existing systems or generate terraform/ansible. 
+    web interface, to provide a portable generic json file, that can be used to visualise existing systems or generate terraform/ansible.
 
+
+## Blogs
+- [Introduction to OKIT the OCI Designer Toolkit](https://www.ateam-oracle.com/introduction-to-okit-the-oci-designer-toolkit)
+- [The OCI Designer Toolkit Templates Feature](https://www.ateam-oracle.com/the-oci-designer-toolkit-templates-feature)
+- [The OCI Designer Toolkit Query Feature](https://www.ateam-oracle.com/the-oci-designer-toolkit-query-feature)
+- [OCI Designer Toolkit Resource Manager Integration](https://www.ateam-oracle.com/oci-designer-toolkit-resource-manager-integration)
+- [The OCI Designer Toolkit Documentation Generation](https://www.ateam-oracle.com/the-oci-designer-toolkit-documentation-generation)
 
 
 ## Installation
@@ -75,7 +83,7 @@ Upload the generated __oci_api_key_public.pem__ to OCI through the [console](htt
 openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key.pem | openssl md5 -c
 ```
 
-##### Config File
+##### OCI Config File
 
 Create the OCI cli __config__ file in the directory __<USER HOME DIR>/.oci__ with contents similar to that below.
 The __*key_file*__ is a fixed value because the contents of the __<USER HOME DIR>/.oci__ will be mounted to the
@@ -90,10 +98,30 @@ tenancy=ocid1.tenancy.oc1..aaaaaaaawpqblfem........
 region=us-phoenix-1
 ```
 
+##### GIT Settings File
+
+If Git integration is required you will need to create a __git_repositories__ file within the directory 
+__<USER HOME DIR>/.oci__ with contents similar to that below.
+
+```properties
+[OKIT Community]
+branch = master
+url = git@url1.git
+
+[Internal]
+branch = BRANCHNAME
+url = git@url2.git
+```
+
+This properties file contains a list of the Git repositories you want to access. It assumes that you are using public/private
+key access and the key files exist within your __<USER HOME DIR>/.ssh__ directory and the __<USER HOME DIR>/.ssh/config__
+defines the key/url mapping.
+
+
 #### Run Container
 
 ```bash
-docker run -d --rm -p 80:80 --volume <USER HOME DIR>/okit/user/templates:/okit/templates --volume <USER HOME DIR>/.oci:/root/.oci --name okit okit
+docker run -d --rm -p 80:80 --volume <USER HOME DIR>/okit/user/templates:/okit/templates --volume <USER HOME DIR>/.oci:/root/.oci --volume <USER HOME DIR>/.ssh:/root/.ssh --name okit okit
 ```
 
 Once started the Designer BUI can be accessed on [http://localhost/okit/designer](http://localhost/okit/designer)

containers/cloud/cloud_init.yml

@@ -12,7 +12,7 @@ runcmd:
   # Install Required Packages because the packages section may not complete before the runcmd
   - sudo bash -c "yum install -y git python-oci-cli oci-utils"
   # Install Required Python Modules
-  - sudo bash -c "pip3 install --no-cache-dir flask==1.1.1 gitpython==3.1.11 gunicorn==20.0.4 oci==2.22.0 oci-cli==2.14.1 pandas==1.1.2 python-magic==0.4.18 pyyaml==5.3.1 requests==2.24.0 xlsxwriter==1.3.6"
+  - sudo bash -c "pip3 install --no-cache-dir flask==1.1.1 gitpython==3.1.11 git-url-parse==1.2.2 gunicorn==20.0.4 oci==2.22.0 oci-cli==2.14.1 openpyxl==3.0.5 pandas==1.1.2 python-magic==0.4.18 pyyaml==5.3.1 requests==2.24.0 xlsxwriter==1.3.6"
   # Clone OKIT
   - sudo bash -c "git clone -b master --depth 1 https://github.com/oracle/oci-designer-toolkit.git /okit"
   - sudo bash -c "mkdir /okit/{log,workspace}"

containers/cloud/okit-ws.json

@@ -51,7 +51,7 @@
       },
       "metadata": {
         "ssh_authorized_keys": "",
-        "user_data": "#cloud-config\nwrite_files:\n  # Add aliases to bash (Note: At time of writing the append flag does not appear to be working)\n  - path: /etc/.bashrc\n    append: true\n    content: |\n      alias lh='ls -lash'\n      alias lt='ls -last'\n      alias env='/usr/bin/env | sort'\n\nruncmd:\n  # Install Required Packages because the packages section may not complete before the runcmd\n  - sudo bash -c \"yum install -y git python-oci-cli oci-utils\"\n  # Install Required Python Modules\n  - sudo bash -c \"pip3 install --no-cache-dir flask==1.1.1 gitpython==3.1.11 gunicorn==20.0.4 oci==2.22.0 oci-cli==2.14.1 pandas==1.1.2 python-magic==0.4.18 pyyaml==5.3.1 requests==2.24.0 xlsxwriter==1.3.6\"\n  # Clone OKIT\n  - sudo bash -c \"git clone -b master --depth 1 https://github.com/oracle/oci-designer-toolkit.git /okit\"\n  - sudo bash -c \"mkdir /okit/{log,workspace}\"\n  # Add additional environment information because append does not appear to work in write_file\n  - sudo bash -c \"echo 'source /etc/.bashrc' >> /etc/bashrc\"\n  - sudo bash -c \"echo 'export OCI_CLI_AUTH=instance_principal' >> /etc/bashrc\"\n  - sudo bash -c \"echo 'export OKIT_VM_COMPARTMENT=`oci-metadata -g \"compartmentID\" --value-only`' >> /etc/bashrc\"\n  # Copy GUnicorn Service File\n  - sudo bash -c 'sed \"s/{COMPARTMENT_OCID}/`oci-metadata -g compartmentID --value-only`/\" /okit/containers/services/gunicorn.service > /etc/systemd/system/gunicorn.service'\n  # Enable Gunicorn Service\n  - sudo systemctl enable gunicorn.service\n  - sudo systemctl start gunicorn.service\n\nfinal_message: \"**** The system is finally up, after $UPTIME seconds ****\"\n"
+        "user_data": "#cloud-config\nwrite_files:\n  # Add aliases to bash (Note: At time of writing the append flag does not appear to be working)\n  - path: /etc/.bashrc\n    append: true\n    content: |\n      alias lh='ls -lash'\n      alias lt='ls -last'\n      alias env='/usr/bin/env | sort'\n\nruncmd:\n  # Install Required Packages because the packages section may not complete before the runcmd\n  - sudo bash -c \"yum install -y git python-oci-cli oci-utils\"\n  # Install Required Python Modules\n  - sudo bash -c \"pip3 install --no-cache-dir flask==1.1.1 gitpython==3.1.11 git-url-parse==1.2.2 gunicorn==20.0.4 oci==2.22.0 oci-cli==2.14.1 openpyxl==3.0.5 pandas==1.1.2 python-magic==0.4.18 pyyaml==5.3.1 requests==2.24.0 xlsxwriter==1.3.6\"\n  # Clone OKIT\n  - sudo bash -c \"git clone -b master --depth 1 https://github.com/oracle/oci-designer-toolkit.git /okit\"\n  - sudo bash -c \"mkdir /okit/{log,workspace}\"\n  # Add additional environment information because append does not appear to work in write_file\n  - sudo bash -c \"echo 'source /etc/.bashrc' >> /etc/bashrc\"\n  - sudo bash -c \"echo 'export OCI_CLI_AUTH=instance_principal' >> /etc/bashrc\"\n  - sudo bash -c \"echo 'export OKIT_VM_COMPARTMENT=`oci-metadata -g \"compartmentID\" --value-only`' >> /etc/bashrc\"\n  # Copy GUnicorn Service File\n  - sudo bash -c 'sed \"s/{COMPARTMENT_OCID}/`oci-metadata -g compartmentID --value-only`/\" /okit/containers/services/gunicorn.service > /etc/systemd/system/gunicorn.service'\n  # Enable Gunicorn Service\n  - sudo systemctl enable gunicorn.service\n  - sudo systemctl start gunicorn.service\n\nfinal_message: \"**** The system is finally up, after $UPTIME seconds ****\"\n"
       },
       "block_storage_volume_ids": [],
       "object_storage_bucket_ids": [],

containers/docker/8-slim/Dockerfile

@@ -5,7 +5,7 @@
 FROM oraclelinux:8-slim
 LABEL "provider"="Oracle" \
       "issues"="https://github.com/oracle/oci-designer-toolkit/issues" \
-      "version"="0.15.0" \
+      "version"="0.16.0" \
       "description"="OKIT Web Server Container." \
       "copyright"="Copyright (c) 2020, Oracle and/or its affiliates."
 SHELL ["/bin/bash", "-c"]
@@ -39,8 +39,10 @@ RUN microdnf install -y \
  && pip3 install --no-cache-dir \
         flask==1.1.1 \
         gitpython==3.1.11 \
+        git-url-parse==1.2.2 \
         gunicorn==20.0.4 \
         oci==2.22.0 \
+        openpyxl==3.0.5 \
         pandas==1.1.2 \
         python-magic==0.4.18 \
         pyyaml==5.3.1 \

containers/docker/Dockerfile

@@ -5,7 +5,7 @@
 FROM oraclelinux:7-slim
 LABEL "provider"="Oracle" \
       "issues"="https://github.com/oracle/oci-designer-toolkit/issues" \
-      "version"="0.15.0" \
+      "version"="0.16.0" \
       "description"="OKIT Web Server Container." \
       "copyright"="Copyright (c) 2020, Oracle and/or its affiliates."
 SHELL ["/bin/bash", "-c"]
@@ -44,17 +44,21 @@ RUN yum install -y \
  && pip3 install --no-cache-dir \
         flask==1.1.1 \
         gitpython==3.1.11 \
+        git-url-parse==1.2.2 \
         gunicorn==20.0.4 \
         oci==2.22.0 \
+        openpyxl==3.0.5 \
         pandas==1.1.2 \
         python-magic==0.4.18 \
         pyyaml==5.3.1 \
         requests==2.24.0 \
         xlsxwriter==1.3.6 \
 # Create Workspace
- && mkdir -p /okit/{config,log,visualiser,okitweb,workspace,templates,skeletons} \
+ && mkdir -p /okit/{config,git,log,visualiser,okitweb,workspace,templates,skeletons} \
  && mkdir -p /okit/okitweb/static/okit/templates \
- && ln -s /okit/templates /okit/okitweb/static/okit/templates/user
+ && ln -s /okit/templates/user /okit/okitweb/static/okit/templates/user \
+ && ln -s /okit/git /okit/okitweb/static/okit/templates/git \
+ && ln -s /okit/git /okit/okitweb/static/okit/git
 # Copy source code
 COPY okitweb /okit/okitweb
 COPY visualiser /okit/visualiser

documentation/Installation.md

@@ -73,6 +73,37 @@ Further information on the config file can be found on the OCI sdk page [SDK and
 
 
 
+## Git Connection
+
+### GIT Repositories File
+
+If Git integration is required you will need to create a __git_repositories__ file within the directory
+__<USER HOME DIR>/.oci__ with contents similar to that below.
+
+```properties
+[OKIT Community]
+branch = master
+url = git@url1.git
+
+[Internal]
+branch = BRANCHNAME
+url = git@url2.git
+```
+
+This properties file contains a list of the Git repositories you want to access. It assumes that you are using public/private
+key access and the key files exist within your __<USER HOME DIR>/.ssh__ directory and the __<USER HOME DIR>/.ssh/config__
+defines the key/url mapping.
+
+
+
+
+
+
+
+
+
+
+
 ## Quick Start Runtime
 
 Docker is the recommended runtime container for OKIT and the project contains a top-level Dockerfile to facilitate direct
@@ -86,7 +117,7 @@ docker build --tag okit --force-rm https://github.com/oracle/oci-designer-toolki
 ### Run Container
 
 ```bash
-docker run -d --rm -p 80:80 --volume <USER HOME DIR>/okit/user/templates:/okit/templates --volume <USER HOME DIR>/.oci:/root/.oci --name okit okit
+docker run -d --rm -p 80:80 --volume <USER HOME DIR>/okit/user/templates:/okit/templates --volume <USER HOME DIR>/.oci:/root/.oci --volume <USER HOME DIR>/.ssh:/root/.ssh --name okit okit
 ```
 
 Once started the Designer BUI can be accessed on [http://localhost/okit/designer](http://localhost/okit/designer)
@@ -139,6 +170,13 @@ If you do not have git installed locally the current release of OKIT can be retr
 Before building/rebuilding your chosen container you will need to copy the contents of [&lt;USER HOME DIR&gt;/.oci](#config-file) 
 to the __oci-designer-toolkit/containers/oci__ directory.
 
+#### Create Git Repository Properties File & Copy SSH Keys
+If you decide to use Git as a repository for your templates you will need to create the Git Repository properties file
+[__oci-designer-toolkit/containers/oci/git_repositories__](#git-repositories-file). Once this has been done the contents of
+__&lt;USER HOME DIR&gt;/.ssh__ will need to be copied to __oci-designer-toolkit/containers/ssh__
+
+
+
 #### Docker Compose
 The docker image is the recommended runtime server OKIT provides a simple Docker Compose script to build and start the container.
 
@@ -244,7 +282,7 @@ Once the Instance has been created the following commands will install OKIT and
 # Install Required Packages because the packages section may not complete before the runcmd
 sudo bash -c "yum install -y git python-oci-cli oci-utils"
 # Install Required Python Modules
-sudo bash -c "pip3 install --no-cache-dir flask==1.1.1 gitpython==3.1.11 gunicorn==20.0.4 oci==2.22.0 oci-cli==2.14.1 pandas==1.1.2 python-magic==0.4.18 pyyaml==5.3.1 requests==2.24.0 xlsxwriter==1.3.6"
+sudo bash -c "pip3 install --no-cache-dir flask==1.1.1 gitpython==3.1.11 git-url-parse==1.2.2 gunicorn==20.0.4 oci==2.22.0 oci-cli==2.14.1 pandas==1.1.2 python-magic==0.4.18 pyyaml==5.3.1 requests==2.24.0 xlsxwriter==1.3.6"
 # Clone OKIT
 sudo bash -c "git clone -b master --depth 1 https://github.com/oracle/oci-designer-toolkit.git /okit"
 sudo bash -c "mkdir /okit/{log,workspace}"

okitweb/__init__.py

@@ -32,6 +32,9 @@ def create_app(test_config=None):
     except OSError:
         pass
 
+    # Add Upload location
+    app.config['UPLOADS_FOLDER'] = '/okit/uploads'
+
     # a simple page that says hello
     @app.route('/hello')
     def hello():

okitweb/okitImport.py

@@ -15,9 +15,12 @@
 from flask import Blueprint
 from flask import request
 import json
+from werkzeug.utils import secure_filename
 from common.okitCommon import logJson
 from common.okitLogging import getLogger
 from parsers.okitHclJsonParser import OkitHclJsonParser
+from parsers.okitCceJsonParser import OkitCceJsonParser
+from parsers.OkitCd3ExcelParser import OkitCd3ExcelParser
 
 # Configure logging
 logger = getLogger()
@@ -41,3 +44,38 @@ def parseHclJson():
         return json.dumps(response_json, sort_keys=False, indent=2, separators=(',', ': '))
     else:
         return '404'
+
+@bp.route('ccejson', methods=(['GET']))
+def parseCceJson():
+    #logger.debug('JSON : {0:s}'.format(str(request.json)))
+    if request.method == 'GET':
+        query_string = request.query_string
+        parsed_query_string = urllib.parse.unquote(query_string.decode())
+        query_json = json.loads(parsed_query_string)
+        logJson(query_json)
+        # Import CCE
+        parser = OkitCceJsonParser()
+        response_json = parser.parse(query_json)
+        logJson(response_json)
+        return json.dumps(response_json, sort_keys=False, indent=2, separators=(',', ': '))
+    else:
+        return '404'
+
+@bp.route('cd3xlsx', methods=(['POST']))
+def parseCd3Xlsx():
+    if request.method == 'POST':
+        if 'file' in request.files:
+            file = request.files['file']
+            if file and file.filename != '':
+                filename = os.path.join('/okit/workspace', secure_filename(file.filename))
+                logger.info("Saving Xlsx File {0!s:s}".format(filename))
+                file.save(filename)
+                # Import CD3
+                parser = OkitCd3ExcelParser()
+                response_json = parser.parse(filename)
+                logJson(response_json)
+        else:
+            response_json = {}
+        return json.dumps(response_json, sort_keys=False, indent=2, separators=(',', ': '))
+    else:
+        return '404'