Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic Structure #5

Open
wants to merge 69 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
63590c4
Initial structure
thenakliman Mar 14, 2016
a0a4a1a
Removed .pyc file
thenakliman Mar 14, 2016
7a1ce10
Removed pyc files
thenakliman Mar 14, 2016
f5733a4
correction in files
thenakliman Mar 14, 2016
0d3cf5b
Added basic structure
thenakliman Mar 15, 2016
76e47a7
Changed code structure
thenakliman Mar 16, 2016
33b69b6
Added base module for curl request
karelyatin Mar 17, 2016
4129a5e
Merge pull request #1 from karelyatin/feature_scheduler
karelyatin Mar 17, 2016
6103cd3
Change in structure
thenakliman Mar 17, 2016
0d56ca3
Added structure for Scheduler
thenakliman Mar 21, 2016
b576615
Image create
thenakliman Mar 22, 2016
5c1b300
updated image create
karelyatin Mar 23, 2016
ef6f987
Merge pull request #2 from karelyatin/feature_scheduler
karelyatin Mar 23, 2016
ca1a0f5
Code merged
thenakliman Mar 24, 2016
6614c12
Merge branch 'feature_scheduler' of https://github.com/thenakliman/do…
thenakliman Mar 24, 2016
95642c8
Network and volume
thenakliman Mar 27, 2016
ed966e8
used urllib for post
karelyatin Mar 29, 2016
ee11d02
Merge pull request #3 from karelyatin/feature_scheduler
karelyatin Mar 29, 2016
a578983
Container with post requests
thenakliman Mar 29, 2016
e632caa
added methods network and volume
karelyatin Mar 29, 2016
bd88e15
Merge pull request #4 from karelyatin/feature_scheduler
karelyatin Mar 29, 2016
09b44bd
Changed permission and added some API
thenakliman Apr 2, 2016
26297a8
added methods to images
karelyatin Apr 2, 2016
215a1a5
Changes for the latest api
thenakliman Apr 4, 2016
839f52c
Changes in containers, images, networks, no changes in volumes
thenakliman Apr 23, 2016
1510445
Changes for volumes
thenakliman Apr 24, 2016
3156873
Added functionality for driver of scheduler
thenakliman Apr 24, 2016
ba00f2f
Removed base.py
thenakliman Apr 24, 2016
8959150
Changes for membership
thenakliman Apr 25, 2016
0eff03a
Changes for consul
thenakliman Apr 25, 2016
aa26361
Consul Changes
thenakliman Apr 27, 2016
a8bdcb8
Consul driver added
thenakliman Apr 28, 2016
b2f4425
Added Consul exceptions
thenakliman Apr 28, 2016
562efac
Added dockyard-engine for processing
thenakliman May 8, 2016
cd1e570
Changes in Linux bridge drivers
thenakliman May 13, 2016
6dac2c3
corrections ..
thenakliman May 13, 2016
819b715
Corrections
May 14, 2016
f317a7c
Corrections in implementation
thenakliman May 14, 2016
2e5099c
Modifications
thenakliman May 17, 2016
d7268cb
Corrections in bridge driver
thenakliman May 18, 2016
8faeee7
Added suport for adding routes for the new network
thenakliman May 20, 2016
6efb915
Fixed for issues 20, 24, 25, 27, 28, 29, 31
May 21, 2016
4081893
Patch for Issue 22
May 21, 2016
f5b6a8b
Work on Controller for docker engine
thenakliman May 23, 2016
94ddbe9
API for interface
thenakliman May 25, 2016
b6fe20e
APIs for interface
thenakliman May 26, 2016
d6db3c4
included oslo logging against issue #6
karelyatin May 28, 2016
8fcdf5f
Merge pull request #45 from karelyatin/feature_scheduler
karelyatin May 28, 2016
d53e546
Additing API's for addition of IP address.
May 28, 2016
8b1bcff
Adding preprocessor and post processor for requests
thenakliman May 29, 2016
d206f34
Resolved merge conflicts
thenakliman May 29, 2016
6a2260c
Added controllers for dockyard engine.
thenakliman May 30, 2016
1e8a311
updated for dockyard engine APIs
thenakliman May 31, 2016
adb61a3
Corrections for attaching floating ip API
thenakliman Jun 1, 2016
b1159c2
restructure of the code.
thenakliman Jun 5, 2016
b25a26c
Added check for consul servics
thenakliman Jun 10, 2016
3e31fb6
Changes in consul driver
thenakliman Jun 14, 2016
e14fcfb
Changes for consul to return only up nodes
thenakliman Jun 14, 2016
36fc8a3
Fixed issue #10
thenakliman Jun 18, 2016
e72898f
Fixes for Providing storage capablity
thenakliman Jun 19, 2016
bdb4806
Changes for consul as database
thenakliman Jun 22, 2016
c56e01d
Problem removed for wait time
Jun 28, 2016
407fc4b
Added code again
thenakliman Jun 28, 2016
99af59d
Scheduling on more than on node
thenakliman Jun 29, 2016
883fe7f
Adding logic for preprocessing and post processing
thenakliman Jul 2, 2016
92ea031
Added support for synchronization
thenakliman Jul 5, 2016
800e5d9
Added synchronization for all objects
thenakliman Jul 5, 2016
9dfb7f1
added change for storing only IDs of resources
thenakliman Jul 6, 2016
cc1a0f9
To optimize speed of earch converted list to dict
thenakliman Jul 6, 2016
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*.pyc
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
recursive-include public *
3 changes: 3 additions & 0 deletions bugs/fixed_bugs.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
=================================================================
Add fixed bugs in this file. Select a format to add bugs.
=================================================================
3 changes: 3 additions & 0 deletions bugs/implemented_blueprints.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
========================================================================
Add already implemented.
========================================================================
3 changes: 3 additions & 0 deletions bugs/known_bugs.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
=================================================================
Add known bugs in this file. Select a format to add bugs.
=================================================================
3 changes: 3 additions & 0 deletions bugs/new_blueprints.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
========================================================================
Add blueprints that needs to implmented.
========================================================================
Empty file added dockyard/__init__.py
Empty file.
Empty file added dockyard/api/__init__.py
Empty file.
42 changes: 42 additions & 0 deletions dockyard/api/app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
import pecan

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct importing sequence as per standard given in pep8.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other style checks defined by pep8 should also be added and tested.

from dockyard.api import config as api_config
from oslo_config import cfg
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think some requirement file also need to be created which specify the dependency modules(ex. oslo_config) for running dockyard.

from pecan import make_app

from dockyard import model


API_SERVICE_OPT = [
cfg.PortOpt('port',
default=5869,
help='Port for the dockyard service.'),
cfg.IPOpt('host',
default='127.0.0.1',
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This address should be 0.0.0.0 so that anyone from remote machine is able to communicate to this server.

help='Listening address for dockyard service'),
]

CONF = cfg.CONF
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure but there must be a way to define values of these parameters through configuration file. So that these default values can be overridden. 💯

opt_group = cfg.OptGroup(name='default',
title='Group for the default values for dockyard api')
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

values for dockyard ---> values of dockyard

CONF.register_group(opt_group)
CONF.register_opts(API_SERVICE_OPT, opt_group)
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Provide a sample file also for the reference, so that it can be used for development purpose.


def get_pecan_config():
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the purpose of this function, we have to identify and then remove if not required.

# Set up the pecan configuration
filename = api_config.__file__.replace('.pyc', '.py')
return pecan.configuration.conf_from_file(filename)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think specific import can be done here, from pecan import configuration, need to check any guideline, use better and consistent.


def setup_app(config=None):

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Write documentation for the each class and method of the class or function.

if not config:
config = get_pecan_config()

model.init_model()
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not being used but it will be useful, when databases will be initialized at the start of the application.

app_conf = dict(config.app)

return make_app(
app_conf.pop('root'),
logging=getattr(config, 'logging', {}),
**app_conf
)
24 changes: 24 additions & 0 deletions dockyard/api/config.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Pecan Application Configurations
app = {
'root': 'dockyard.controllers.root.RootController',
'modules': ['dockyard', 'dockyard.api'],
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per documentation, for modules key values should be one. You can have two values but recommended is one.
It's should be in the same directory, where your app.py file is located.

Therefor correct it and put only one required value.

'static_root': '%(confdir)s/public',
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per documentation,

The directory where your static files can be found (relative to the project root). Pecan comes with middleware that can be used to serve static files (like CSS and Javascript files) during development.

i think, for out project static files are not required therefor, these it needs to be considered, whether we should keep it or not.

'template_path': '%(confdir)s/dockyard/templates',
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per documentation:
Points to the directory where your template files live (relative to the project root).

Discuss and make sure that it should be removed if not required in this project.

'debug': True,
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is ok right now but in production environment, it should not be in the debug mode.

'errors': {
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check all the options after this is really being used in the project.

404: '/error/404',
'__force_dict__': True
},
'acl_public_routes': [
'/',
'/v1',
],

}

# Custom Configurations must be in Python dictionary format::
#
# foo = {'bar':'baz'}
#
# All configurations are accessible at::
# pecan.conf
Empty file added dockyard/cmd/__init__.py
Empty file.
15 changes: 15 additions & 0 deletions dockyard/cmd/api.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
from wsgiref import simple_server
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is ok to use this server but we should atleast know, why this server is being used not any other server. It is better, we do a bit of research on it. It will help us in understanding about servers.


from oslo_config import cfg
from oslo_log import log as logging
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logging not used in this module

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently application is running in foreground, we need to make it run as a service.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

from dockyard.api import app as api_app

def main():
app = api_app.setup_app()

# create the wsgi server and start it
host, port = cfg.CONF.default.host, cfg.CONF.default.port
srv = simple_server.make_server(host, port, app)
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have to decide about loggin polices, As logging is not being done now.

srv.serve_forever()

main()
Empty file added dockyard/common/__init__.py
Empty file.
244 changes: 244 additions & 0 deletions dockyard/common/base.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,244 @@
import sys
import io
import json

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

base.py can be removed if not used

from oslo_log import log as logging

# python 2 and python 3 compatibility library
from six import iteritems

try:
import urllib3
except ImportError:
raise ImportError('python client requires urllib3')

try:
# for python3
from urllib.parse import urlencode
except ImportError:
# for python2
from urllib import urlencode


logger = logging.getLogger(__name__)


class RESTResponse(io.IOBase):

def __init__(self, resp):
self.urllib3_response = resp
self.status = resp.status
self.reason = resp.reason
self.data = resp.data

def getheaders(self):
"""
Returns a dictionary of the response headers.
"""
return self.urllib3_response.getheaders()

def getheader(self, name, default=None):
"""
Returns a given response header.
"""
return self.urllib3_response.getheader(name, default)


class RESTClientObject(object):

def __init__(self, pools_size=4):
# http pool manager
self.pool_manager = urllib3.PoolManager(
num_pools=pools_size
)

def agent(self, url):
"""
Use `urllib3.util.parse_url` for backward compatibility.
Return proper pool manager for the http/https schemes.
"""
url = urllib3.util.parse_url(url)
scheme = url.scheme
if scheme == 'https':
return self.ssl_pool_manager
else:
return self.pool_manager

def request(self, method, url, query_params=None, headers=None,
body=None, post_params=None):
"""
:param method: http request method
:param url: http request url
:param query_params: query parameters in the url
:param headers: http request headers
:param body: request json body, for `application/json`
:param post_params: request post parameters,
`application/x-www-form-urlencode`
and `multipart/form-data`
"""
method = method.upper()
assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', 'PATCH']

if post_params and body:
raise ValueError(
"body parameter cannot be used with post_params parameter."
)

post_params = post_params or {}
headers = headers or {}

if 'Content-Type' not in headers:
headers['Content-Type'] = 'application/json'

# For `POST`, `PUT`, `PATCH`
if method in ['POST', 'PUT', 'PATCH']:
if query_params:
url += '?' + urlencode(query_params)
if headers['Content-Type'] == 'application/json':
response = self.agent(url).request(method, url,
body=json.dumps(body),
headers=headers)
if headers['Content-Type'] == 'application/x-www-form-urlencoded':
response = self.agent(url).request(method, url,
encode_multipart=False,
headers=headers)
if headers['Content-Type'] == 'multipart/form-data':
# must del headers['Content-Type'], or the correct Content-Type
# which generated by urllib3 will be overwritten.
del headers['Content-Type']
response = self.agent(url).request(method, url,
encode_multipart=True,
headers=headers)
# For `GET`, `HEAD`, `DELETE`
else:
response = self.agent(url).request(method, url,
headers=headers)
response = RESTResponse(response)

# In the python 3, the response.data is bytes.
# we need to decode it to string.
if sys.version_info > (3,):
response.data = response.data.decode('utf8')
# log response body
#logger.debug("response body: %s" % response.data)

if response.status not in range(200, 206):
raise ApiException(http_resp=response)

return response

def GET(self, url, headers=None, query_params=None):
return self.request("GET", url,
headers=headers,
query_params=query_params)

def HEAD(self, url, headers=None, query_params=None):
return self.request("HEAD", url,
headers=headers,
query_params=query_params)

def DELETE(self, url, headers=None, query_params=None):
return self.request("DELETE", url,
headers=headers,
query_params=query_params)

def POST(self, url, headers=None, query_params=None, post_params=None, body=None):
return self.request("POST", url,
headers=headers,
query_params=query_params,
post_params=post_params,
body=body)

def PUT(self, url, headers=None, query_params=None, post_params=None, body=None):
return self.request("PUT", url,
headers=headers,
query_params=query_params,
post_params=post_params,
body=body)

def PATCH(self, url, headers=None, query_params=None, post_params=None, body=None):
return self.request("PATCH", url,
headers=headers,
query_params=query_params,
post_params=post_params,
body=body)


class ApiException(Exception):

def __init__(self, status=None, reason=None, http_resp=None):
if http_resp:
self.status = http_resp.status
self.reason = http_resp.reason
self.body = http_resp.data
self.headers = http_resp.getheaders()
else:
self.status = status
self.reason = reason
self.body = None
self.headers = None

def __str__(self):
"""
Custom error messages for exception
"""
error_message = "({0})\n"\
"Reason: {1}\n".format(self.status, self.reason)
if self.headers:
error_message += "HTTP response headers: {0}\n".format(self.headers)

if self.body:
error_message += "HTTP response body: {0}\n".format(self.body)

return error_message

class RESTClient(object):
"""
A class with methods to perform JSON requests.
"""

def __init__(self):
self.IMPL = RESTClientObject()

def request(self, *n, **kw):
"""
Perform a REST request and parse the response.
"""
return self.IMPL.request(*n, **kw)

def GET(self, *n, **kw):
"""
Perform a GET request using `RESTClient.request()`.
"""
return self.IMPL.GET(*n, **kw)

def HEAD(self, *n, **kw):
"""
Perform a HEAD request using `RESTClient.request()`.
"""
return self.IMPL.GET(*n, **kw)

def POST(self, *n, **kw):
"""
Perform a POST request using `RESTClient.request()`
"""
return self.IMPL.POST(*n, **kw)

def PUT(self, *n, **kw):
"""
Perform a PUT request using `RESTClient.request()`
"""
return self.IMPL.PUT(*n, **kw)

def PATCH(self, *n, **kw):
"""
Perform a PATCH request using `RESTClient.request()`
"""
return self.IMPL.PATCH(*n, **kw)

def DELETE(self, *n, **kw):
"""
Perform a DELETE request using `RESTClient.request()`
"""
return self.IMPL.DELETE(*n, **kw)

Empty file.
14 changes: 14 additions & 0 deletions dockyard/common/cluster/cluster.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
from oslo_config import cfg

class Cluster(object):
def __init__(self):
pass

def register(self, cluster_id, host_ip, port):
pass

def unregister(self, cluster_id, host_ip):
pass

def get_hosts(self):
return cfg.CONF.membership.hosts
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Write a better membership algorithm because this one always adds its own host only.

We have to decide about membership protocol to be used in this product.

Empty file.
Empty file.
Loading