From efb732c9c5ba670e10cb7edbdf584649f4604094 Mon Sep 17 00:00:00 2001 From: Matthew Richards Date: Wed, 21 Oct 2020 13:28:28 +0000 Subject: [PATCH 1/3] #134: Make file adhere to Black's 88 character/line rule - Since the Python files in the repo adhere to this limit, it seems sensible that the README file also following this formatting rule where possible --- README.md | 93 +++++++++++++++++++++++++++++++------------------------ 1 file changed, 52 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 600507d7..ee5b7cfb 100644 --- a/README.md +++ b/README.md @@ -17,10 +17,9 @@ ICAT API to interface with the Data Gateway - [Running Tests](#running-tests) - - ## Requirements -All requirements can be installed with `pip install -r requirements.txt`, and all development requirements can be installed with `pip install -r dev-requirements.txt` +All requirements can be installed with `pip install -r requirements.txt`, and all +development requirements can be installed with `pip install -r dev-requirements.txt` The required python libraries: - [SQLAlchemy](https://www.sqlalchemy.org/) @@ -30,17 +29,20 @@ The required python libraries: - [pip-tools](https://github.com/jazzband/pip-tools) (For generating requirements.txt) ## Setup and running the API -The database connection needs to be set up first. This is set in config.json, an example config file called `config.json.example` is provided. +The database connection needs to be set up first. This is set in config.json, an example +config file called `config.json.example` is provided. Ideally the API would be run with: `python -m src.main` However it can be run with the flask run command as shown below: -**Warning: the host, port and debug config options will not be respected when the API is run this way** +**Warning: the host, port and debug config options will not be respected when the API is +run this way** -To use `flask run`, the enviroment variable `FLASK_APP` should be set to `src/main.py`. Once this is -set the API can be run with `flask run` while inside the root directory of the project. The `flask run` command gets installed with flask. +To use `flask run`, the enviroment variable `FLASK_APP` should be set to `src/main.py`. +Once this is set the API can be run with `flask run` while inside the root directory of +the project. The `flask run` command gets installed with flask. Examples shown: Unix @@ -69,10 +71,10 @@ e.g. `http://localhost:5000/sessions` - ## Project structure -The project consists of 3 main packages: common, src and test. common contains modules shared across test and src such as the database mapping classes. -src contains the api resources and their http method definitions, and test contains tests for each endpoint. +The project consists of 3 main packages: common, src and test. common contains modules +shared across test and src such as the database mapping classes. src contains the api +resources and their http method definitions, and test contains tests for each endpoint. This is illustrated below. @@ -118,59 +120,67 @@ This is illustrated below. └── config.json ````` #### Main -`main.py` is where the flask_restful api is set up. This is where each endpoint resource class is generated and mapped -to an endpoint. +`main.py` is where the flask_restful api is set up. This is where each endpoint resource +class is generated and mapped to an endpoint. Example: - `api.add_resource(get_endpoint(entity_name, endpoints[entity_name]), f"/{entity_name.lower()}")` +```python +api.add_resource(get_endpoint(entity_name, endpoints[entity_name]), f"/{entity_name.lower()}") +``` #### Endpoints -The logic for each endpoint are within `/src/resources`. They are split into entities, non_entities and -table_endpoints. The entities package contains `entities_map` which maps entity names to their sqlalchemy -model. The `entity_endpoint` module contains the function that is used to generate endpoints at start up. -`table_endpoints` contains the endpoint classes that are table specific. Finally, non_entities contains the -session endpoint. +The logic for each endpoint are within `/src/resources`. They are split into entities, +non_entities and table_endpoints. The entities package contains `entities_map` which +maps entity names to their sqlalchemy model. The `entity_endpoint` module contains the +function that is used to generate endpoints at start up. `table_endpoints` contains the +endpoint classes that are table specific. Finally, non_entities contains the session +endpoint. #### Mapped classes -The classes mapped from the database are stored in `/common/database/models.py`. Each model was -automatically generated using sqlacodegen. A class `EntityHelper` is defined so that each model may -inherit two methods `to_dict()` and `update_from_dict(dictionary)`, both used for returning entities -and updating them, in a form easily converted to JSON. - - +The classes mapped from the database are stored in `/common/database/models.py`. Each +model was automatically generated using sqlacodegen. A class `EntityHelper` is defined +so that each model may inherit two methods `to_dict()` and +`update_from_dict(dictionary)`, both used for returning entities and updating them, in a +form easily converted to JSON. ## Database Generator -There is a tool to generate mock data into the database. It is located in `util/icat_db_generator.py` -By default it will generate 20 years worth of data (approx 70,000 entities). The script makes use of -`random` and `Faker` and is seeded with a seed of 1. The seed and number of years of data generated can -be changed by using the arg flags `-s` or `--seed` for the seed, and `-y` or `--years` for the number of years. -For example: -`python -m util.icat_db_generator -s 4 -y 10` Would set the seed to 4 and generate 10 years of data. +There is a tool to generate mock data into the database. It is located in +`util/icat_db_generator.py`. By default it will generate 20 years worth of data (approx +70,000 entities). The script makes use of `random` and `Faker` and is seeded with a seed +of 1. The seed and number of years of data generated can be changed by using the arg +flags `-s` or `--seed` for the seed, and `-y` or `--years` for the number of years. For +example: `python -m util.icat_db_generator -s 4 -y 10` Would set the seed to 4 and +generate 10 years of data. #### Querying and filtering: -The querying and filtering logic is located in `/common/database_helpers.py`. In this module the abstract `Query` and -`QueryFilter` classes are defined as well as their implementations. The functions that are used by various endpoints to -query the database are also in this module. -Class diagrams for this module: +The querying and filtering logic is located in `/common/database_helpers.py`. In this +module the abstract `Query` and `QueryFilter` classes are defined as well as their +implementations. The functions that are used by various endpoints to query the database +are also in this module. + + +#### Class diagrams for this module: ![image](https://user-images.githubusercontent.com/44777678/67954353-ba69ef80-fbe8-11e9-81e3-0668cea3fa35.png) ![image](https://user-images.githubusercontent.com/44777678/67954834-7fb48700-fbe9-11e9-96f3-ffefc7277ebd.png) #### Authentication -Each request requires a valid session ID to be provided in the Authorization header. This header should take the form of `{"Authorization":"Bearer "}` A session ID can be obtained by -sending a post request to `/sessions/` -All endpoint methods that require a session id are decorated with `@requires_session_id` +Each request requires a valid session ID to be provided in the Authorization header. +This header should take the form of `{"Authorization":"Bearer "}` A session +ID can be obtained by sending a post request to `/sessions/`. All endpoint methods that +require a session id are decorated with `@requires_session_id` #### Generating the swagger spec: `openapi.yaml` -The swagger generation script is located in `/src/swagger/swagger_generator.py`. The script will only run when -the config option `generate_swagger` is set to true in `config.json`. The generator decorates the first endpoint -resource class in it's module to get the name of the entity. It then creates the correct paths using the name of the +The swagger generation script is located in `/src/swagger/swagger_generator.py`. The +script will only run when the config option `generate_swagger` is set to true in +`config.json`. The generator decorates the first endpoint resource class in it's module +to get the name of the entity. It then creates the correct paths using the name of the entity and outputs the swagger spec to `openapi.yaml` Example of the decorator: @@ -191,6 +201,7 @@ To run the tests use `python -m unittest discover` When writing code for this repository, [Black](https://black.readthedocs.io/en/stable/) is used as the code linter/formatter to ensure the code is kept Pythonic. Installing the dev requirements will ensure this package is installed. This repository uses the -default settings for Black; to use, execute the following command on the root directory of this repo: +default settings for Black; to use, execute the following command on the root directory +of this repo: `black .` From c2e8099f00861e3c61cf945415091ecc03131c3b Mon Sep 17 00:00:00 2001 From: Matthew Richards Date: Wed, 21 Oct 2020 15:45:08 +0000 Subject: [PATCH 2/3] #134: Edit Swagger doc section of README to reflect changes in generation --- README.md | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index ee5b7cfb..d67fceec 100644 --- a/README.md +++ b/README.md @@ -175,23 +175,22 @@ ID can be obtained by sending a post request to `/sessions/`. All endpoint metho require a session id are decorated with `@requires_session_id` - #### Generating the swagger spec: `openapi.yaml` -The swagger generation script is located in `/src/swagger/swagger_generator.py`. The -script will only run when the config option `generate_swagger` is set to true in -`config.json`. The generator decorates the first endpoint resource class in it's module -to get the name of the entity. It then creates the correct paths using the name of the -entity and outputs the swagger spec to `openapi.yaml` - -Example of the decorator: -```python -@swagger_gen.resource_wrapper() -class DataCollectionDatasets(Resource): - @requires_session_id - @queries_records - def get(self): - return get_rows_by_filter(DATACOLLECTIONDATASET, get_filters_from_query_string()), 200 -``` +When the config option `generate_swagger` is set to true in `config.json`, a YAML +file defining the API using OpenAPI standards will be created at +`src/swagger/openapi.yaml`. [apispec](https://apispec.readthedocs.io/en/latest/) is used +to help with this, with an `APISpec()` object created in `src/main.py` which is added to +(using `APISpec.path()`) when the endpoints are created for Flask. These paths are +iterated over and ordered alphabetically, to ensure `openapi.yaml` only changes if there +have been changes to the Swagger docs of the API; without that code, Git will detect +changes on that file everytime startup occurs (preventing a clean development repo). The +contents of the `APISpec` object are written to a YAML file and is used when the user +goes to the configured (root) page in their browser. + +The endpoint related files in `src/resources/` contain `__doc__` which have the Swagger +docs for each type of endpoint. `src/resources/swagger/` contain code to aid Swagger doc +generation, with a plugin (`RestfulPlugin`) created for `apispec` to extract Swagger +documentation from `flask-restful` functions. ## Running Tests From da21adb735c842b72da6bcb4e6806e286f974786 Mon Sep 17 00:00:00 2001 From: Matthew Richards Date: Wed, 21 Oct 2020 15:49:14 +0000 Subject: [PATCH 3/3] #134: Misc. changes of existing sections in README - Also updated `requirements.in` (with a freshly compiled `requirements.txt`) to reflect the requirement of Python ICAT --- README.md | 12 +++++++++--- requirements.in | 2 +- requirements.txt | 2 +- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index d67fceec..da152ef1 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,8 @@ ICAT API to interface with the Data Gateway - [Main](#main) - [Endpoints](#endpoints) - [Mapped classes](#mapped-classes) + - [Database Generator](#database-generator) + - [Class Diagrams](#class-diagrams-for-this-module) - [Querying and filtering](#querying-and-filtering) - [Swagger Generation](#generating-the-swagger-spec-openapiyaml) - [Authentication](#authentication) @@ -25,8 +27,12 @@ The required python libraries: - [SQLAlchemy](https://www.sqlalchemy.org/) - [flask-restful](https://github.com/flask-restful/flask-restful/) - [pymysql](https://pymysql.readthedocs.io/en/latest/) - - [pyyaml](https://pyyaml.org/wiki/PyYAMLDocumentation) (For the swagger generation) - - [pip-tools](https://github.com/jazzband/pip-tools) (For generating requirements.txt) + - [apispec](https://apispec.readthedocs.io/en/latest/) (For the swagger generation) + - [pip-tools](https://github.com/jazzband/pip-tools) (For generating + requirements.txt) + - [python-icat](https://python-icat.readthedocs.io/en/stable) (For ICAT backend) + + ## Setup and running the API The database connection needs to be set up first. This is set in config.json, an example @@ -196,7 +202,7 @@ documentation from `flask-restful` functions. ## Running Tests To run the tests use `python -m unittest discover` -## Linter +## Code Formatter When writing code for this repository, [Black](https://black.readthedocs.io/en/stable/) is used as the code linter/formatter to ensure the code is kept Pythonic. Installing the dev requirements will ensure this package is installed. This repository uses the diff --git a/requirements.in b/requirements.in index 8b8d42d8..1db77744 100644 --- a/requirements.in +++ b/requirements.in @@ -4,4 +4,4 @@ pymysql == 0.9.3 flask-cors == 3.0.8 apispec == 3.3.0 flask-swagger-ui == 3.25.0 -pyyaml == 5.1.2 \ No newline at end of file +python-icat == 0.17.0 \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index 752e3265..f6055a6c 100644 --- a/requirements.txt +++ b/requirements.txt @@ -15,8 +15,8 @@ itsdangerous==1.1.0 # via flask jinja2==2.10.1 # via flask markupsafe==1.1.1 # via jinja2 pymysql==0.9.3 # via -r requirements.in +python-icat==0.17.0 # via -r requirements.in pytz==2019.2 # via flask-restful -pyyaml==5.1.2 # via -r requirements.in six==1.12.0 # via flask-cors, flask-restful sqlalchemy==1.3.8 # via -r requirements.in werkzeug==0.16.0 # via flask