Merge branch 'feature/fix-code-linting-#184' into feature/test-multip…

…le-backends-#150

README.md

@@ -10,6 +10,8 @@ ICAT API to interface with the Data Gateway
       - [Main](#main)
       - [Endpoints](#endpoints)
       - [Mapped classes](#mapped-classes)
+      - [Database Generator](#database-generator)
+      - [Class Diagrams](#class-diagrams-for-this-module)
       - [Querying and filtering](#querying-and-filtering)
       - [Swagger Generation](#generating-the-swagger-spec-openapiyaml)
       - [Authentication](#authentication)
@@ -241,10 +243,12 @@ Ideally, the API would be run with:
 However it can be run with the flask run command as shown below:
 
 
-**Warning: the host, port and debug config options will not be respected when the API is run this way**
+**Warning: the host, port and debug config options will not be respected when the API is
+run this way**
 
-To use `flask run`, the enviroment variable `FLASK_APP` should be set to `src/main.py`. Once this is
-set the API can be run with `flask run` while inside the root directory of the project. The `flask run` command gets installed with flask.
+To use `flask run`, the enviroment variable `FLASK_APP` should be set to `src/main.py`.
+Once this is set the API can be run with `flask run` while inside the root directory of
+the project. The `flask run` command gets installed with flask.
 
 Examples shown:
 Unix
@@ -268,10 +272,10 @@ More information can be found [here](http://flask.pocoo.org/docs/1.0/cli/).
 By default the api will run on `http://localhost:5000` and all requests are made here
 e.g. `http://localhost:5000/sessions`
 
-
 ## Project structure
-The project consists of 3 main packages: common, src and test. common contains modules shared across test and src such as the database mapping classes.
-src contains the api resources and their http method definitions, and test contains tests for each endpoint.
+The project consists of 3 main packages: common, src and test. common contains modules
+shared across test and src such as the database mapping classes. src contains the api
+resources and their http method definitions, and test contains tests for each endpoint.
 
 This is illustrated below.
 
@@ -317,67 +321,76 @@ This is illustrated below.
     └── config.json
  `````
 #### Main
-`main.py` is where the flask_restful api is set up. This is where each endpoint resource class is generated and mapped
-to an endpoint.
+`main.py` is where the flask_restful api is set up. This is where each endpoint resource
+class is generated and mapped to an endpoint.
 
 Example:
- `api.add_resource(get_endpoint(entity_name, endpoints[entity_name]), f"/{entity_name.lower()}")`
+```python
+api.add_resource(get_endpoint(entity_name, endpoints[entity_name]), f"/{entity_name.lower()}")
+```
 
 
 #### Endpoints
-The logic for each endpoint are within `/src/resources`. They are split into entities, non_entities and
-table_endpoints. The entities package contains `entities_map` which maps entity names to their sqlalchemy
-model. The `entity_endpoint` module contains the function that is used to generate endpoints at start up.
-`table_endpoints` contains the endpoint classes that are table specific. Finally, non_entities contains the
-session endpoint.
+The logic for each endpoint are within `/src/resources`. They are split into entities,
+non_entities and table_endpoints. The entities package contains `entities_map` which
+maps entity names to their sqlalchemy model. The `entity_endpoint` module contains the
+function that is used to generate endpoints at start up. `table_endpoints` contains the
+endpoint classes that are table specific. Finally, non_entities contains the session
+endpoint.
 
 
 #### Mapped classes
-The classes mapped from the database are stored in `/common/database/models.py`. Each model was
-automatically generated using sqlacodegen. A class `EntityHelper` is defined so that each model may
-inherit two methods `to_dict()` and `update_from_dict(dictionary)`, both used for returning entities
-and updating them, in a form easily converted to JSON.
+The classes mapped from the database are stored in `/common/database/models.py`. Each
+model was automatically generated using sqlacodegen. A class `EntityHelper` is defined
+so that each model may inherit two methods `to_dict()` and
+`update_from_dict(dictionary)`, both used for returning entities and updating them, in a
+form easily converted to JSON.
 
 
 ## Database Generator
-There is a tool to generate mock data into the database. It is located in `util/icat_db_generator.py`
-By default it will generate 20 years worth of data (approx 70,000 entities). The script makes use of
-`random` and `Faker` and is seeded with a seed of 1. The seed and number of years of data generated can
-be changed by using the arg flags `-s` or `--seed` for the seed, and `-y` or `--years` for the number of years.
-For example:
-`python -m util.icat_db_generator -s 4 -y 10` Would set the seed to 4 and generate 10 years of data.
+There is a tool to generate mock data into the database. It is located in
+`util/icat_db_generator.py`. By default it will generate 20 years worth of data (approx
+70,000 entities). The script makes use of `random` and `Faker` and is seeded with a seed
+of 1. The seed and number of years of data generated can be changed by using the arg
+flags `-s` or `--seed` for the seed, and `-y` or `--years` for the number of years. For
+example: `python -m util.icat_db_generator -s 4 -y 10` Would set the seed to 4 and
+generate 10 years of data.
 
 
 #### Querying and filtering:
-The querying and filtering logic is located in `/common/database_helpers.py`. In this module the abstract `Query` and
-`QueryFilter` classes are defined as well as their implementations. The functions that are used by various endpoints to
-query the database are also in this module.
-Class diagrams for this module:
+The querying and filtering logic is located in `/common/database_helpers.py`. In this
+module the abstract `Query` and `QueryFilter` classes are defined as well as their
+implementations. The functions that are used by various endpoints to query the database
+are also in this module.
+
+
+#### Class diagrams for this module:
 ![image](https://user-images.githubusercontent.com/44777678/67954353-ba69ef80-fbe8-11e9-81e3-0668cea3fa35.png)
 ![image](https://user-images.githubusercontent.com/44777678/67954834-7fb48700-fbe9-11e9-96f3-ffefc7277ebd.png)
 
 
 #### Authentication
-Each request requires a valid session ID to be provided in the Authorization header. This header should take the form of `{"Authorization":"Bearer <session_id>"}` A session ID can be obtained by
-sending a post request to `/sessions/`
-All endpoint methods that require a session id are decorated with `@requires_session_id`
-
+Each request requires a valid session ID to be provided in the Authorization header.
+This header should take the form of `{"Authorization":"Bearer <session_id>"}` A session
+ID can be obtained by sending a post request to `/sessions/`. All endpoint methods that
+require a session id are decorated with `@requires_session_id`
 
 #### Generating the swagger spec: `openapi.yaml`
-The swagger generation script is located in `/src/swagger/swagger_generator.py`. The script will only run when
-the config option `generate_swagger` is set to true in `config.json`. The generator decorates the first endpoint
-resource class in it's module to get the name of the entity. It then creates the correct paths using the name of the
-entity and outputs the swagger spec to `openapi.yaml`
-
-Example of the decorator:
-```python
-@swagger_gen.resource_wrapper()
-class DataCollectionDatasets(Resource):
-    @requires_session_id
-    @queries_records
-    def get(self):
-        return get_rows_by_filter(DATACOLLECTIONDATASET, get_filters_from_query_string()), 200
-```
+When the config option `generate_swagger` is set to true in `config.json`, a YAML
+file defining the API using OpenAPI standards will be created at
+`src/swagger/openapi.yaml`. [apispec](https://apispec.readthedocs.io/en/latest/) is used
+to help with this, with an `APISpec()` object created in `src/main.py` which is added to
+(using `APISpec.path()`) when the endpoints are created for Flask. These paths are
+iterated over and ordered alphabetically, to ensure `openapi.yaml` only changes if there
+have been changes to the Swagger docs of the API; without that code, Git will detect
+changes on that file everytime startup occurs (preventing a clean development repo). The
+contents of the `APISpec` object are written to a YAML file and is used when the user
+goes to the configured (root) page in their browser.
+
+The endpoint related files in `src/resources/` contain `__doc__` which have the Swagger
+docs for each type of endpoint. `src/resources/swagger/` contain code to aid Swagger doc
+generation, with a plugin (`RestfulPlugin`) created for `apispec` to extract Swagger
+documentation from `flask-restful` functions.
 
 
 ## Running Tests

datagateway_api/common/database/filters.py

@@ -73,6 +73,8 @@ def apply_filter(self, query):
 
         if self.operation == "eq":
             query.base_query = query.base_query.filter(field == self.value)
+        elif self.operation == "ne":
+            query.base_query = query.base_query.filter(field != self.value)
         elif self.operation == "like":
             query.base_query = query.base_query.filter(field.like(f"%{self.value}%"))
         elif self.operation == "lt":

datagateway_api/src/main.py

@@ -10,7 +10,6 @@
 
 from datagateway_api.common.backends import create_backend
 from datagateway_api.common.config import config
-from datagateway_api.common.exceptions import ApiError
 from datagateway_api.common.logger_setup import setup_logger
 from datagateway_api.src.resources.entities.entity_endpoint import (
     get_count_endpoint,
@@ -38,6 +37,16 @@
 app = Flask(__name__)
 
 
+class CustomErrorHandledApi(Api):
+    """
+    This class overrides `handle_error` function from the API class from `flask_restful`
+    to correctly return response codes and exception messages from uncaught exceptions
+    """
+
+    def handle_error(self, e):
+        return str(e), e.status_code
+
+
 def create_app_infrastructure(flask_app):
     swaggerui_blueprint = get_swaggerui_blueprint(
         "", "/openapi.json", config={"app_name": "DataGateway API OpenAPI Spec"},
@@ -53,9 +62,7 @@ def create_app_infrastructure(flask_app):
 
     CORS(flask_app)
     flask_app.url_map.strict_slashes = False
-    api = Api(flask_app)
-
-    flask_app.register_error_handler(ApiError, handle_error)
+    api = CustomErrorHandledApi(flask_app)
 
     initialise_spec(spec)
 
@@ -144,10 +151,11 @@ def create_api_endpoints(flask_app, api, spec):
 def openapi_config(spec):
     # Reorder paths (e.g. get, patch, post) so openapi.yaml only changes when there's a
     # change to the Swagger docs, rather than changing on each startup
-    log.debug("Reordering OpenAPI docs to alphabetical order")
-    for entity_data in spec._paths.values():
-        for endpoint_name in sorted(entity_data.keys()):
-            entity_data.move_to_end(endpoint_name)
+    if config.is_generate_swagger():
+        log.debug("Reordering OpenAPI docs to alphabetical order")
+        for entity_data in spec._paths.values():
+            for endpoint_name in sorted(entity_data.keys()):
+                entity_data.move_to_end(endpoint_name)
 
     openapi_spec_path = Path(__file__).parent / "swagger/openapi.yaml"
     with open(openapi_spec_path, "w") as f:

datagateway_api/src/resources/entities/entity_endpoint.py

@@ -194,7 +194,7 @@ def get(self, id_):
             parameters:
                 - in: path
                   required: true
-                  name: id
+                  name: id_
                   description: The id of the entity to retrieve
                   schema:
                     type: integer
@@ -229,7 +229,7 @@ def delete(self, id_):
             parameters:
                 - in: path
                   required: true
-                  name: id
+                  name: id_
                   description: The id of the entity to delete
                   schema:
                     type: integer
@@ -261,7 +261,7 @@ def patch(self, id_):
             parameters:
                 - in: path
                   required: true
-                  name: id
+                  name: id_
                   description: The id of the entity to update
                   schema:
                     type: integer

datagateway_api/src/swagger/initialise_spec.py

@@ -26,7 +26,9 @@ def initialise_spec(spec):
             "in": "query",
             "name": "where",
             "description": "Apply where filters to the query. The possible operators"
-            " are like, gte, lte, in and eq",
+            " are: ne, like, lt, lte, gt, gte, in and eq. Please modify the examples"
+            " before executing a request if you are having issues with the example"
+            " values.",
             "schema": {
                 "type": "array",
                 "items": {
@@ -54,9 +56,28 @@ def initialise_spec(spec):
                             },
                             {
                                 "type": "object",
-                                "title": "Greater than or equal",
+                                "title": "Inequality",
                                 "properties": {
-                                    "gte": {
+                                    "ne": {
+                                        "oneOf": [
+                                            {"type": "string"},
+                                            {"type": "number"},
+                                            {"type": "integer"},
+                                            {"type": "boolean"},
+                                        ]
+                                    }
+                                },
+                            },
+                            {
+                                "type": "object",
+                                "title": "Substring equality",
+                                "properties": {"like": {"type": "string"}},
+                            },
+                            {
+                                "type": "object",
+                                "title": "Less than",
+                                "properties": {
+                                    "lt": {
                                         "oneOf": [
                                             {"type": "number"},
                                             {"type": "integer"},
@@ -78,8 +99,27 @@ def initialise_spec(spec):
                             },
                             {
                                 "type": "object",
-                                "title": "Substring equality",
-                                "properties": {"like": {"type": "string"}},
+                                "title": "Greater than",
+                                "properties": {
+                                    "gt": {
+                                        "oneOf": [
+                                            {"type": "number"},
+                                            {"type": "integer"},
+                                        ]
+                                    }
+                                },
+                            },
+                            {
+                                "type": "object",
+                                "title": "Greater than or equal",
+                                "properties": {
+                                    "gte": {
+                                        "oneOf": [
+                                            {"type": "number"},
+                                            {"type": "integer"},
+                                        ]
+                                    }
+                                },
                             },
                             {
                                 "type": "object",
@@ -102,11 +142,14 @@ def initialise_spec(spec):
                 },
             },
             "examples": {
-                "eq": {"value": [{"ID": {"eq": 1}}]},
-                "like": {"value": [{"NAME": {"like": "dog"}}]},
-                "gte": {"value": [{"ID": {"gte": 50}}]},
-                "lte": {"value": [{"ID": {"lte": 50}}]},
-                "in": {"value": [{"ID": {"in": [1, 2, 3]}}]},
+                "eq": {"value": [{"id": {"eq": 1}}]},
+                "ne": {"value": [{"id": {"ne": 1}}]},
+                "like": {"value": [{"name": {"like": "dog"}}]},
+                "lt": {"value": [{"id": {"lt": 10}}]},
+                "lte": {"value": [{"id": {"lte": 50}}]},
+                "gt": {"value": [{"id": {"gt": 10}}]},
+                "gte": {"value": [{"id": {"gte": 50}}]},
+                "in": {"value": [{"id": {"in": [1, 2, 3]}}]},
             },
         },
     )
@@ -120,7 +163,7 @@ def initialise_spec(spec):
             "description": "Apply order filters to the query. Given a field and"
             " direction, order the returned entities.",
             "schema": {"type": "array", "items": {"type": "string"}},
-            "examples": {"asc": {"value": ["ID asc"]}, "desc": {"value": ["ID desc"]}},
+            "examples": {"asc": {"value": ["id asc"]}, "desc": {"value": ["id desc"]}},
         },
     )