Merge pull request #179 from ral-facilities/feature/update-swagger-ge…

…naration-in-readme-#134

Update Swagger Generation Section in README.md

README.md

@@ -10,37 +10,45 @@ ICAT API to interface with the Data Gateway
       - [Main](#main)
       - [Endpoints](#endpoints)
       - [Mapped classes](#mapped-classes)
+      - [Database Generator](#database-generator)
+      - [Class Diagrams](#class-diagrams-for-this-module)
       - [Querying and filtering](#querying-and-filtering)
       - [Swagger Generation](#generating-the-swagger-spec-openapiyaml)
       - [Authentication](#authentication)
   - [Database Generator](#database-generator)
   - [Running Tests](#running-tests)
 
 
-
-
 ## Requirements
-All requirements can be installed with `pip install -r requirements.txt`, and all development requirements can be installed with `pip install -r dev-requirements.txt`
+All requirements can be installed with `pip install -r requirements.txt`, and all
+development requirements can be installed with `pip install -r dev-requirements.txt`
 
 The required python libraries:  
    - [SQLAlchemy](https://www.sqlalchemy.org/)    
    - [flask-restful](https://github.com/flask-restful/flask-restful/)  
    - [pymysql](https://pymysql.readthedocs.io/en/latest/)  
-   - [pyyaml](https://pyyaml.org/wiki/PyYAMLDocumentation) (For the swagger generation)
-   - [pip-tools](https://github.com/jazzband/pip-tools) (For generating requirements.txt)
+   - [apispec](https://apispec.readthedocs.io/en/latest/) (For the swagger generation)
+   - [pip-tools](https://github.com/jazzband/pip-tools) (For generating
+    requirements.txt)
+   - [python-icat](https://python-icat.readthedocs.io/en/stable) (For ICAT backend)
+
+
 
 ## Setup and running the API   
-The database connection needs to be set up first. This is set in config.json, an example config file called `config.json.example` is provided.
+The database connection needs to be set up first. This is set in config.json, an example
+config file called `config.json.example` is provided.
 
 Ideally the API would be run with:  
 `python -m src.main`
 However it can be run with the flask run command as shown below:
 
 
-**Warning: the host, port and debug config options will not be respected when the API is run this way**
+**Warning: the host, port and debug config options will not be respected when the API is 
+run this way**
 
-To use `flask run`, the enviroment variable `FLASK_APP` should be set to `src/main.py`. Once this is 
-set the API can be run with `flask run` while inside the root directory of the project. The `flask run` command gets installed with flask.   
+To use `flask run`, the enviroment variable `FLASK_APP` should be set to `src/main.py`.
+Once this is set the API can be run with `flask run` while inside the root directory of
+the project. The `flask run` command gets installed with flask.
 
 Examples shown:  
 Unix
@@ -69,10 +77,10 @@ e.g.
 `http://localhost:5000/sessions`
 
 
-
 ## Project structure
-The project consists of 3 main packages: common, src and test. common contains modules shared across test and src such as the database mapping classes.
-src contains the api resources and their http method definitions, and test contains tests for each endpoint.
+The project consists of 3 main packages: common, src and test. common contains modules
+shared across test and src such as the database mapping classes. src contains the api
+resources and their http method definitions, and test contains tests for each endpoint.
 
 This is illustrated below.
 
@@ -118,79 +126,87 @@ This is illustrated below.
     └── config.json
  `````
 #### Main
-`main.py` is where the flask_restful api is set up. This is where each endpoint resource class is generated and mapped 
-to an endpoint.
+`main.py` is where the flask_restful api is set up. This is where each endpoint resource
+class is generated and mapped to an endpoint.
 
 Example:  
- `api.add_resource(get_endpoint(entity_name, endpoints[entity_name]), f"/{entity_name.lower()}")`	   
+```python
+api.add_resource(get_endpoint(entity_name, endpoints[entity_name]), f"/{entity_name.lower()}")
+```
 
 
 #### Endpoints
-The logic for each endpoint are within `/src/resources`. They are split into entities, non_entities and 
-table_endpoints. The entities package contains `entities_map` which maps entity names to their sqlalchemy
-model. The `entity_endpoint` module contains the function that is used to generate endpoints at start up.
-`table_endpoints` contains the endpoint classes that are table specific. Finally, non_entities contains the
-session endpoint.
+The logic for each endpoint are within `/src/resources`. They are split into entities,
+non_entities and table_endpoints. The entities package contains `entities_map` which
+maps entity names to their sqlalchemy model. The `entity_endpoint` module contains the
+function that is used to generate endpoints at start up. `table_endpoints` contains the
+endpoint classes that are table specific. Finally, non_entities contains the session
+endpoint.
 
 
 #### Mapped classes
-The classes mapped from the database are stored in `/common/database/models.py`. Each model was 
-automatically generated using sqlacodegen. A class `EntityHelper` is defined so that each model may
-inherit two methods `to_dict()` and `update_from_dict(dictionary)`, both used for returning entities 
-and updating them, in a form easily converted to JSON.  
-
-
+The classes mapped from the database are stored in `/common/database/models.py`. Each
+model was automatically generated using sqlacodegen. A class `EntityHelper` is defined
+so that each model may inherit two methods `to_dict()` and 
+`update_from_dict(dictionary)`, both used for returning entities and updating them, in a
+form easily converted to JSON.  
 
 
 ## Database Generator
-There is a tool to generate mock data into the database. It is located in `util/icat_db_generator.py`
-By default it will generate 20 years worth of data (approx 70,000 entities). The script makes use of 
-`random` and `Faker` and is seeded with a seed of 1. The seed and number of years of data generated can 
-be changed by using the arg flags `-s` or `--seed` for the seed, and `-y` or `--years` for the number of years.
-For example:  
-`python -m util.icat_db_generator -s 4 -y 10` Would set the seed to 4 and generate 10 years of data.
+There is a tool to generate mock data into the database. It is located in
+`util/icat_db_generator.py`. By default it will generate 20 years worth of data (approx
+70,000 entities). The script makes use of `random` and `Faker` and is seeded with a seed
+of 1. The seed and number of years of data generated can be changed by using the arg
+flags `-s` or `--seed` for the seed, and `-y` or `--years` for the number of years. For
+example: `python -m util.icat_db_generator -s 4 -y 10` Would set the seed to 4 and
+generate 10 years of data.
 
 
 #### Querying and filtering:
-The querying and filtering logic is located in `/common/database_helpers.py`. In this module the abstract `Query` and
-`QueryFilter` classes are defined as well as their implementations. The functions that are used by various endpoints to
-query the database are also in this module.
-Class diagrams for this module:
+The querying and filtering logic is located in `/common/database_helpers.py`. In this
+module the abstract `Query` and `QueryFilter` classes are defined as well as their
+implementations. The functions that are used by various endpoints to query the database
+are also in this module.
+
+
+#### Class diagrams for this module:
 ![image](https://user-images.githubusercontent.com/44777678/67954353-ba69ef80-fbe8-11e9-81e3-0668cea3fa35.png)  
 ![image](https://user-images.githubusercontent.com/44777678/67954834-7fb48700-fbe9-11e9-96f3-ffefc7277ebd.png)
 
 
 #### Authentication
-Each request requires a valid session ID to be provided in the Authorization header. This header should take the form of `{"Authorization":"Bearer <session_id>"}` A session ID can be obtained by
-sending a post request to `/sessions/`  
-All endpoint methods that require a session id are decorated with `@requires_session_id`
-
+Each request requires a valid session ID to be provided in the Authorization header.
+This header should take the form of `{"Authorization":"Bearer <session_id>"}` A session
+ID can be obtained by sending a post request to `/sessions/`. All endpoint methods that
+require a session id are decorated with `@requires_session_id`
 
 
 #### Generating the swagger spec: `openapi.yaml`
-The swagger generation script is located in `/src/swagger/swagger_generator.py`. The script will only run when
-the config option `generate_swagger` is set to true in `config.json`. The generator decorates the first endpoint
-resource class in it's module to get the name of the entity. It then creates the correct paths using the name of the 
-entity and outputs the swagger spec to `openapi.yaml` 
-
-Example of the decorator:
-```python
-@swagger_gen.resource_wrapper()
-class DataCollectionDatasets(Resource):
-    @requires_session_id
-    @queries_records
-    def get(self):
-        return get_rows_by_filter(DATACOLLECTIONDATASET, get_filters_from_query_string()), 200
-```
+When the config option `generate_swagger` is set to true in `config.json`, a YAML
+file defining the API using OpenAPI standards will be created at
+`src/swagger/openapi.yaml`. [apispec](https://apispec.readthedocs.io/en/latest/) is used
+to help with this, with an `APISpec()` object created in `src/main.py` which is added to
+(using `APISpec.path()`) when the endpoints are created for Flask. These paths are 
+iterated over and ordered alphabetically, to ensure `openapi.yaml` only changes if there
+have been changes to the Swagger docs of the API; without that code, Git will detect
+changes on that file everytime startup occurs (preventing a clean development repo). The
+contents of the `APISpec` object are written to a YAML file and is used when the user
+goes to the configured (root) page in their browser.
+
+The endpoint related files in `src/resources/` contain `__doc__` which have the Swagger
+docs for each type of endpoint. `src/resources/swagger/` contain code to aid Swagger doc
+generation, with a plugin (`RestfulPlugin`) created for `apispec` to extract Swagger
+documentation from `flask-restful` functions.
 
 
 ## Running Tests
 To run the tests use `python -m unittest discover`
 
-## Linter
+## Code Formatter
 When writing code for this repository, [Black](https://black.readthedocs.io/en/stable/)
 is used as the code linter/formatter to ensure the code is kept Pythonic. Installing
 the dev requirements will ensure this package is installed. This repository uses the
-default settings for Black; to use, execute the following command on the root directory of this repo:
+default settings for Black; to use, execute the following command on the root directory
+of this repo:
 
 `black .`

requirements.in

@@ -4,4 +4,4 @@ pymysql == 0.9.3
 flask-cors == 3.0.8
 apispec == 3.3.0
 flask-swagger-ui == 3.25.0
-pyyaml == 5.1.2
+python-icat == 0.17.0

requirements.txt

@@ -15,8 +15,8 @@ itsdangerous==1.1.0       # via flask
 jinja2==2.10.1            # via flask
 markupsafe==1.1.1         # via jinja2
 pymysql==0.9.3            # via -r requirements.in
+python-icat==0.17.0       # via -r requirements.in
 pytz==2019.2              # via flask-restful
-pyyaml==5.1.2             # via -r requirements.in
 six==1.12.0               # via flask-cors, flask-restful
 sqlalchemy==1.3.8         # via -r requirements.in
 werkzeug==0.16.0          # via flask