Merge pull request #153 from ral-facilities/feature/python-icat-where…

…-filter-#142

Change structure of Filters and Implement Basic WHERE Filter for Python ICAT backend

README.md

@@ -7,9 +7,9 @@ ICAT API to interface with the Data Gateway
   - [Requirements](#requirements)
   - [Setup and running the API](#setup-and-running-the-api)
   - [Project structure](#project-structure)
-      - [Main:](#main)
-      - [Endpoints:](#endpoints)
-      - [Mapped classes:](#mapped-classes)
+      - [Main](#main)
+      - [Endpoints](#endpoints)
+      - [Mapped classes](#mapped-classes)
       - [Querying and filtering](#querying-and-filtering)
       - [Swagger Generation](#generating-the-swagger-spec-openapiyaml)
       - [Authentication](#authentication)
@@ -26,7 +26,6 @@ The required python libraries:
    - [SQLAlchemy](https://www.sqlalchemy.org/)    
    - [flask-restful](https://github.com/flask-restful/flask-restful/)  
    - [pymysql](https://pymysql.readthedocs.io/en/latest/)  
-   - [requests](https://2.python-requests.org/en/master/)
    - [pyyaml](https://pyyaml.org/wiki/PyYAMLDocumentation) (For the swagger generation)
    - [pip-tools](https://github.com/jazzband/pip-tools) (For generating requirements.txt)
 
@@ -80,12 +79,18 @@ This is illustrated below.
 
 `````
 ─── datagateway-api
-    ├── common  
+    ├── common
+    │   ├── database
+    │   │   ├── backend.py
+    │   │   ├── filters.py
+    │   │   └── helpers.py
+    │   ├── icat
     │   ├── models
     │   │   └── db_models.py
+    │   ├── backends.py
     │   ├── constants.py
-    │   ├── database_helpers.py
     │   ├── exceptions.py
+    │   ├── filters.py
     │   └── helpers.py
     ├── src
     │   ├── resources
@@ -112,23 +117,23 @@ This is illustrated below.
     ├── logs.log
     └── config.json
  `````
-#### Main:
+#### Main
 `main.py` is where the flask_restful api is set up. This is where each endpoint resource class is generated and mapped 
 to an endpoint.
 
 Example:  
  `api.add_resource(get_endpoint(entity_name, endpoints[entity_name]), f"/{entity_name.lower()}")`	   
 
 
-#### Endpoints:  
+#### Endpoints
 The logic for each endpoint are within `/src/resources`. They are split into entities, non_entities and 
 table_endpoints. The entities package contains `entities_map` which maps entity names to their sqlalchemy
 model. The `entity_endpoint` module contains the function that is used to generate endpoints at start up.
 `table_endpoints` contains the endpoint classes that are table specific. Finally, non_entities contains the
 session endpoint.
 
 
-#### Mapped classes:
+#### Mapped classes
 The classes mapped from the database are stored in `/common/models/db_models.py`. Each model was 
 automatically generated using sqlacodegen. A class `EntityHelper` is defined so that each model may
 inherit two methods `to_dict()` and `update_from_dict(dictionary)`, both used for returning entities 
@@ -182,4 +187,10 @@ class DataCollectionDatasets(Resource):
 ## Running Tests
 To run the tests use `python -m unittest discover`
 
+## Linter
+When writing code for this repository, [Black](https://black.readthedocs.io/en/stable/)
+is used as the code linter/formatter to ensure the code is kept Pythonic. Installing
+the dev requirements will ensure this package is installed. This repository uses the
+default settings for Black; to use, execute the following command on the root directory of this repo:
 
+`black .`

common/backend.py

@@ -44,7 +44,9 @@ def logout(self, session_id):
     @abstractmethod
     def get_with_filters(self, session_id, entity_type, filters):
         """
-        Given a list of filters supplied in json format, returns entities that match the filters for the given entity type
+        Given a list of filters supplied in json format, returns entities that match the
+        filters for the given entity type
+
         :param session_id: The session id of the requesting user
         :param entity_type: The type of entity
         :param filters: The list of filters to be applied
@@ -55,7 +57,9 @@ def get_with_filters(self, session_id, entity_type, filters):
     @abstractmethod
     def create(self, session_id, entity_type, data):
         """
-        Create one or more entities, from the given list containing json. Each entity must not contain its ID
+        Create one or more entities, from the given list containing json. Each entity
+        must not contain its ID
+
         :param session_id: The session id of the requesting user
         :param entity_type: The type of entity
         :param data: The entities to be created
@@ -66,7 +70,9 @@ def create(self, session_id, entity_type, data):
     @abstractmethod
     def update(self, session_id, entity_type, data):
         """
-        Update one or more entities, from the given list containing json. Each entity must contain its ID
+        Update one or more entities, from the given list containing json. Each entity
+        must contain its ID
+
         :param session_id: The session id of the requesting user
         :param entity_type: The type of entity
         :param data: the list of updated values or a dictionary
@@ -77,7 +83,8 @@ def update(self, session_id, entity_type, data):
     @abstractmethod
     def get_one_with_filters(self, session_id, entity_type, filters):
         """
-        returns the first entity that matches a given filter, for a given entity type
+        Returns the first entity that matches a given filter, for a given entity type
+
         :param session_id: The session id of the requesting user
         :param entity_type: The type of entity
         :param filters: the filter to be applied to the query
@@ -88,7 +95,9 @@ def get_one_with_filters(self, session_id, entity_type, filters):
     @abstractmethod
     def count_with_filters(self, session_id, entity_type, filters):
         """
-        returns the count of the entities that match a given filter for a given entity type
+        Returns the count of the entities that match a given filter for a given entity
+        type
+
         :param session_id: The session id of the requesting user
         :param entity_type: The type of entity
         :param filters: the filters to be applied to the query
@@ -100,6 +109,7 @@ def count_with_filters(self, session_id, entity_type, filters):
     def get_with_id(self, session_id, entity_type, id_):
         """
         Gets the entity matching the given ID for the given entity type
+
         :param session_id: The session id of the requesting user
         :param entity_type: The type of entity
         :param id_: the id of the record to find
@@ -111,6 +121,7 @@ def get_with_id(self, session_id, entity_type, id_):
     def delete_with_id(self, session_id, entity_type, id_):
         """
         Deletes the row matching the given ID for the given entity type
+
         :param session_id: The session id of the requesting user
         :param table: the table to be searched
         :param id_: the id of the record to delete
@@ -121,6 +132,7 @@ def delete_with_id(self, session_id, entity_type, id_):
     def update_with_id(self, session_id, entity_type, id_, data):
         """
         Updates the row matching the given ID for the given entity type
+
         :param session_id: The session id of the requesting user
         :param entity_type: The type of entity
         :param id_: the id of the record to update
@@ -130,9 +142,13 @@ def update_with_id(self, session_id, entity_type, id_, data):
         pass
 
     @abstractmethod
-    def get_instrument_facilitycycles_with_filters(self, session_id, instrument_id, filters):
+    def get_instrument_facilitycycles_with_filters(
+        self, session_id, instrument_id, filters
+    ):
         """
-        Given an instrument_id get facility cycles where the instrument has investigations that occur within that cycle
+        Given an instrument_id get facility cycles where the instrument has
+        investigations that occur within that cycle
+
         :param session_id: The session id of the requesting user
         :param filters: The filters to be applied to the query
         :param instrument_id: The id of the instrument
@@ -141,10 +157,13 @@ def get_instrument_facilitycycles_with_filters(self, session_id, instrument_id,
         pass
 
     @abstractmethod
-    def count_instrument_facilitycycles_with_filters(self, session_id, instrument_id, filters):
+    def count_instrument_facilitycycles_with_filters(
+        self, session_id, instrument_id, filters
+    ):
         """
-        Given an instrument_id get the facility cycles count where the instrument has investigations that occur within
-        that cycle
+        Given an instrument_id get the facility cycles count where the instrument has
+        investigations that occur within that cycle
+
         :param session_id: The session id of the requesting user
         :param filters: The filters to be applied to the query
         :param instrument_id: The id of the instrument
@@ -153,9 +172,13 @@ def count_instrument_facilitycycles_with_filters(self, session_id, instrument_id
         pass
 
     @abstractmethod
-    def get_instrument_facilitycycle_investigations_with_filters(self, session_id, instrument_id, facilitycycle_id, filters):
+    def get_instrument_facilitycycle_investigations_with_filters(
+        self, session_id, instrument_id, facilitycycle_id, filters
+    ):
         """
-        Given an instrument id and facility cycle id, get investigations that use the given instrument in the given cycle
+        Given an instrument id and facility cycle id, get investigations that use the
+        given instrument in the given cycle
+
         :param session_id: The session id of the requesting user
         :param filters: The filters to be applied to the query
         :param instrument_id: The id of the instrument
@@ -165,10 +188,13 @@ def get_instrument_facilitycycle_investigations_with_filters(self, session_id, i
         pass
 
     @abstractmethod
-    def count_instrument_facilitycycles_investigations_with_filters(self, session_id, instrument_id, facilitycycle_id, filters):
+    def count_instrument_facilitycycles_investigations_with_filters(
+        self, session_id, instrument_id, facilitycycle_id, filters
+    ):
         """
-        Given an instrument id and facility cycle id, get the count of the investigations that use the given instrument in
-        the given cycle
+        Given an instrument id and facility cycle id, get the count of the
+        investigations that use the given instrument in the given cycle
+        
         :param session_id: The session id of the requesting user
         :param filters: The filters to be applied to the query
         :param instrument_id: The id of the instrument

common/backends.py

@@ -1,5 +1,5 @@
-from common.database_backend import DatabaseBackend
-from common.python_icat_backend import PythonICATBackend
+from common.database.backend import DatabaseBackend
+from common.icat.backend import PythonICATBackend
 from common.backend import Backend
 from common.config import config
 import sys
@@ -11,6 +11,5 @@
 elif backend_type == "python_icat":
     backend = PythonICATBackend()
 else:
-    sys.exit(
-        f"Invalid config value '{backend_type}' for config option backend")
+    sys.exit(f"Invalid config value '{backend_type}' for config option backend")
     backend = Backend()

common/config.py

@@ -1,10 +1,13 @@
 import json
 import sys
 from pathlib import Path
+import requests
+import logging
 
+log = logging.getLogger()
 
-class Config(object):
 
+class Config(object):
     def __init__(self):
         config_path = Path(__file__).parent.parent / "config.json"
         with open(config_path) as target:
@@ -35,7 +38,7 @@ def get_icat_check_cert(self):
             return self.config["icat_check_cert"]
         except:
             # This could be set to true if there's no value, and log a warning
-            # that no value has been found from the config - save app from 
+            # that no value has been found from the config - save app from
             # exiting
             sys.exit("Missing config value, icat_check_cert")
 
@@ -69,5 +72,17 @@ def get_port(self):
         except:
             sys.exit("Missing config value, port")
 
+    def get_icat_properties(self):
+        """
+        ICAT properties can be retrieved using Python ICAT's client object, however this
+        requires the client object to be authenticated which may not always be the case
+        when requesting these properties, hence a HTTP request is sent as an alternative
+        """
+        properties_url = f"{config.get_icat_url()}/icat/properties"
+        r = requests.request("GET", properties_url, verify=config.get_icat_check_cert())
+        icat_properties = r.json()
+
+        return icat_properties
+
 
 config = Config()

common/constants.py

@@ -4,3 +4,4 @@
 class Constants:
     DATABASE_URL = config.get_db_url()
     ACCEPTED_DATE_FORMAT = "%Y-%m-%d %H:%M:%S"
+    ICAT_PROPERTIES = config.get_icat_properties()

common/database_backend.py → common/database/backend.py

@@ -1,18 +1,31 @@
 from common.backend import Backend
-from common.database_helpers import get_facility_cycles_for_instrument, get_facility_cycles_for_instrument_count, \
-    get_investigations_for_instrument_in_facility_cycle, get_investigations_for_instrument_in_facility_cycle_count, \
-    get_rows_by_filter, create_rows_from_json, patch_entities, get_row_by_id, insert_row_into_table, \
-    delete_row_by_id, update_row_from_id, get_filtered_row_count, get_first_filtered_row
-from common.database_helpers import requires_session_id
+from common.database.helpers import (
+    get_facility_cycles_for_instrument,
+    get_facility_cycles_for_instrument_count,
+    get_investigations_for_instrument_in_facility_cycle,
+    get_investigations_for_instrument_in_facility_cycle_count,
+    get_rows_by_filter,
+    create_rows_from_json,
+    patch_entities,
+    get_row_by_id,
+    insert_row_into_table,
+    delete_row_by_id,
+    update_row_from_id,
+    get_filtered_row_count,
+    get_first_filtered_row,
+    requires_session_id,
+)
 from common.helpers import queries_records
 from common.models.db_models import SESSION
 import uuid
 from common.exceptions import AuthenticationError
 import datetime
 
 import logging
+
 log = logging.getLogger()
 
+
 class DatabaseBackend(Backend):
     """
     Class that contains functions to access and modify data in an ICAT database directly
@@ -21,8 +34,14 @@ class DatabaseBackend(Backend):
     def login(self, credentials):
         if credentials["username"] == "user" and credentials["password"] == "password":
             session_id = str(uuid.uuid1())
-            insert_row_into_table(SESSION, SESSION(ID=session_id, USERNAME=f"{credentials['mechanism']}/root",
-                                                   EXPIREDATETIME=datetime.datetime.now() + datetime.timedelta(days=1)))
+            insert_row_into_table(
+                SESSION,
+                SESSION(
+                    ID=session_id,
+                    USERNAME=f"{credentials['mechanism']}/root",
+                    EXPIREDATETIME=datetime.datetime.now() + datetime.timedelta(days=1),
+                ),
+            )
             return session_id
         else:
             raise AuthenticationError("Username and password are incorrect")
@@ -82,20 +101,32 @@ def update_with_id(self, session_id, table, id_, data):
 
     @requires_session_id
     @queries_records
-    def get_instrument_facilitycycles_with_filters(self, session_id, instrument_id, filters):
+    def get_instrument_facilitycycles_with_filters(
+        self, session_id, instrument_id, filters
+    ):
         return get_facility_cycles_for_instrument(instrument_id, filters)
 
     @requires_session_id
     @queries_records
-    def count_instrument_facilitycycles_with_filters(self, session_id, instrument_id, filters):
+    def count_instrument_facilitycycles_with_filters(
+        self, session_id, instrument_id, filters
+    ):
         return get_facility_cycles_for_instrument_count(instrument_id, filters)
 
     @requires_session_id
     @queries_records
-    def get_instrument_facilitycycle_investigations_with_filters(self, session_id, instrument_id, facilitycycle_id, filters):
-        return get_investigations_for_instrument_in_facility_cycle(instrument_id, facilitycycle_id, filters)
+    def get_instrument_facilitycycle_investigations_with_filters(
+        self, session_id, instrument_id, facilitycycle_id, filters
+    ):
+        return get_investigations_for_instrument_in_facility_cycle(
+            instrument_id, facilitycycle_id, filters
+        )
 
     @requires_session_id
     @queries_records
-    def count_instrument_facilitycycles_investigations_with_filters(self, session_id, instrument_id, facilitycycle_id, filters):
-        return get_investigations_for_instrument_in_facility_cycle_count(instrument_id, facilitycycle_id, filters)
+    def count_instrument_facilitycycles_investigations_with_filters(
+        self, session_id, instrument_id, facilitycycle_id, filters
+    ):
+        return get_investigations_for_instrument_in_facility_cycle_count(
+            instrument_id, facilitycycle_id, filters
+        )